• Turn on log4php.debug library usage, in file config.performance.php

  'LOG4PHP_DEBUG' => true,

• Set the log level, in file log4php.properties

  log4php.rootLogger = DEBUG, A1

• The log files will be written in the logs directory. For the rootLogger it will be logs/vtigercrm.log

• Turn on debugging for ADODB instanced in include/database/PearDatabase (at end of file)

  if(empty($adb)) {
   $adb = new PearDatabase();
   $adb->connect();
   // ADD THIS LINE
   $adb->setDebug(true);
  }

• In file, Smarty_setup.php add the following to class constructor function

  function vtigerCRM_Smarty() {
   $this->Smarty();
   $this->template_dir = 'Smarty/templates';
   // ...
   // ADD THIS LINE
   $this->debugging = true;
  }

• Chrome and Firefox have webdevelopers tools and firebug.

• What I usually do is edit the curl call and dump the raw result coming from coreBOS. Obviously the log4php system is also very useful in this case.
• The exact code change is one of these two depending on POST or GET calls:

diff --git a/vtwsclib/Net/curl_http_client.php b/vtwsclib/Net/curl_http_client.php
index c52dd76..a42eb3c 100644
--- a/vtwsclib/Net/curl_http_client.php
+++ b/vtwsclib/Net/curl_http_client.php
@@ -183,7 +183,7 @@ class Curl_HTTP_Client
 
 		//and finally send curl request
 		$result = curl_exec($this->ch);
-
+var_dump($result); // GET
 		if($this->has_error())
 		{
 			return false;
@@ -243,7 +243,7 @@ class Curl_HTTP_Client
 
 		//and finally send curl request
 		$result = curl_exec($this->ch);
-
+var_dump($result);  // POST
 		if($this->has_error())
 		{
 			return false;

Also, the ajax call in the browser can be seen and is very useful also to see the raw result (Firebug is your friend)

Have a read here about how coreBOS outgoing email server works and how to configure it.

To debug the outgoing email server the best is to activate PHPMailer debuging with the changes below and look in the errmail.log file that will be created in the root of your install.

diff --git a/modules/Emails/class.phpmailer.php b/modules/Emails/class.phpmailer.php
index 56271f9..3877204 100644
--- a/modules/Emails/class.phpmailer.php
+++ b/modules/Emails/class.phpmailer.php
@@ -303,7 +303,7 @@ class PHPMailer
      * @type int
      * @see SMTP::$do_debug
      */
-    public $SMTPDebug = 0;
+    public $SMTPDebug = 2;
 
     /**
      * The function/method to use for debugging output.
@@ -311,7 +311,7 @@ class PHPMailer
      * @type string
      * @see SMTP::$Debugoutput
      */
-    public $Debugoutput = "echo";
+    public $Debugoutput = "error_log";
 
     /**
      * Whether to keep SMTP connection open after each message.
@@ -623,7 +623,7 @@ class PHPMailer
         }
         switch ($this->Debugoutput) {
             case 'error_log':
-                error_log($str);
+                error_log($str,3,'errmail.log');
                 break;
             case 'html':
                 //Cleans up output a bit for a better looking display that's HTML-safe
diff --git a/modules/Emails/class.smtp.php b/modules/Emails/class.smtp.php
index e6b4522..286b202 100644
--- a/modules/Emails/class.smtp.php
+++ b/modules/Emails/class.smtp.php
@@ -77,14 +77,14 @@ class SMTP
      * Options: 0 for no output, 1 for commands, 2 for data and commands
      * @type int
      */
-    public $do_debug = 0;
+    public $do_debug = 1;
 
     /**
      * The function/method to use for debugging output.
      * Options: 'echo', 'html' or 'error_log'
      * @type string
      */
-    public $Debugoutput = 'echo';
+    public $Debugoutput = 'error_log';
 
     /**
      * Whether to use VERP.
@@ -151,7 +151,7 @@ class SMTP
         switch ($this->Debugoutput) {
             case 'error_log':
                 //Don't output, just log
-                error_log($str);
+                error_log($str,3,'errmail.log');
                 break;
             case 'html':
                 //Cleans up output a bit for a better looking, HTML-safe output