When your server starts sending unexpected outbound traffic, especially through the httpd process, it's crucial to identify the exact source script responsible. The netstat command shows the connections, but we need to dig deeper into process-level tracing.

Here are three effective methods to pinpoint the originating script:

1. Using lsof for Active Connections

sudo lsof -i TCP:80 -n -P | grep httpd
sudo lsof -p [PID] | grep cwd

2. Strace for Real-time System Calls

sudo strace -f -p [PID] -e trace=network -s 10000
# Alternative for specific IP:
sudo strace -f -p $(pgrep httpd) 2>&1 | grep "connect.*[TARGET_IP]"

3. Combining tcpdump with Process Audit

sudo tcpdump -i any -nn -v host [TARGET_IP] and port 80 -w capture.pcap
# Then analyze with:
sudo auditctl -a exit,always -F arch=b64 -S connect -k http_traffic
sudo ausearch -k http_traffic | grep [TARGET_IP]

For web servers running PHP, add this to your tracing approach:

sudo grep -l [TARGET_IP] /var/log/httpd/*_log
sudo grep -r "fsockopen.*'[TARGET_IP]'" /var/www/
sudo find /var/www/ -name "*.php" -exec grep -l "curl_init.*[TARGET_IP]" {} \;

Create a monitoring script to log suspicious outbound connections:

#!/bin/bash
while true; do
    DATE=$(date +"%Y-%m-%d %H:%M:%S")
    CONNS=$(netstat -tulnp | grep httpd | awk '{print $5}')
    echo "[$DATE] Outbound connections: $CONNS" >> /var/log/httpd_connections.log
    sleep 60
done

For persistent cases, use eBPF tools like bpftrace:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect 
    /comm == "httpd"/ { 
        printf("%s connecting to %s:%d\n", comm, ntop(args->uservaddr->sin_addr), args->uservaddr->sin_port); 
    }'

When you identify the script, check for:

• Malicious code injections
• Compromised third-party libraries
• Legitimate API calls that might be misconfigured
• Cron jobs triggering web requests

---

When monitoring server traffic, you might encounter situations where your Apache (httpd) process is making unexpected outbound connections to specific IP addresses. Here's a comprehensive guide to identify the exact scripts responsible for these connections.

Your netstat -apln | grep httpd output shows multiple established connections to a single IP address. To investigate further, we need to determine which PHP script (or other server-side code) initiated these connections.

First, identify the exact connection details using lsof:

sudo lsof -i -P -n | grep httpd
sudo lsof -p [PID] -P -n

This will show all network connections and the associated process IDs.

For real-time monitoring of system calls, including network connections:

sudo strace -f -p [PID] -e trace=network -s 10000

This will show you all network-related system calls made by the process.

Configure Apache to log complete request information:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" full_trace
CustomLog /var/log/apache2/full_trace.log full_trace

Use tcpdump with advanced filtering to capture HTTP requests:

sudo tcpdump -i any -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

For PHP scripts making external requests, enable the PHP curl log:

// Add to php.ini
curl.log = /var/log/php_curl.log
curl.log_size = 1048576

Alternatively, use this debugging function in your PHP code:

function log_curl_request($ch) {
    $info = curl_getinfo($ch);
    file_put_contents('/tmp/curl_requests.log', 
        date('Y-m-d H:i:s') . " - " . 
        $info['url'] . "\n" . 
        print_r($info, true) . "\n\n", 
        FILE_APPEND);
}

For deep inspection, use systemtap to trace socket operations:

probe kernel.function("sys_socketcall") {
    if (pid() == target()) {
        printf("%s(%d) %s\n", execname(), pid(), probefunc())
    }
}

To monitor currently active connections and their origins:

watch -n 1 'netstat -tnp | grep httpd | awk '\''{print $5 " " $7}'\'' | sort | uniq -c | sort -n'

Create a bash script to log suspicious connections:

#!/bin/bash
while true; do
    DATE=$(date +%Y-%m-%d_%H:%M:%S)
    netstat -tnp | grep httpd | grep ESTABLISHED | awk '{print $5,$7}' | sort | uniq -c | \
    while read count ip pid; do
        if [ $count -gt 5 ]; then
            echo "$DATE - Suspicious connection: $count times to $ip by $pid" >> /var/log/httpd_connections.log
            lsof -p ${pid##*/} | grep cwd >> /var/log/httpd_connections.log
        fi
    done
    sleep 60
done