Yes, you can directly connect two Mellanox ConnectX-2 cards without an InfiniBand switch. You'll need:

• Two machines with Mellanox ConnectX-2 HCAs
• One QSFP (Quad Small Form-factor Pluggable) to QSFP cable (no crossover needed)
• Compatible drivers (mlx4 or mlx5 depending on card variant)

Unlike Ethernet, InfiniBand uses point-to-point connections that don't require crossover cables. A standard QSFP cable with identical pinouts on both ends will work perfectly. For ConnectX-2 cards specifically:

• Use QSFP (not QSFP+) for 20Gb/s connections
• Maximum recommended cable length: 3m for passive copper

After physical connection, configure both machines:

# Load necessary modules
sudo modprobe mlx4_core
sudo modprobe mlx4_ib

# Verify link status
sudo ibstat

# Assign IPoIB addresses (optional for verbs programming)
sudo ip link set ib0 up
sudo ip addr add 192.168.1.1/24 dev ib0  # On first machine
sudo ip addr add 192.168.1.2/24 dev ib0  # On second machine

Here's a basic RDMA write example using libibverbs:

#include 

int main() {
    struct ibv_device **dev_list;
    struct ibv_context *ctx;
    
    // Get device list
    dev_list = ibv_get_device_list(NULL);
    
    // Create context
    ctx = ibv_open_device(dev_list[0]);
    
    // Allocate protection domain
    struct ibv_pd *pd = ibv_alloc_pd(ctx);
    
    // Create completion queue
    struct ibv_cq *cq = ibv_create_cq(ctx, 10, NULL, NULL, 0);
    
    // Register memory region
    char *buf = malloc(4096);
    struct ibv_mr *mr = ibv_reg_mr(pd, buf, 4096, 
        IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE);
    
    // Continue with QP creation and connection setup...
}

If you encounter issues:

1. Check /var/log/messages for Mellanox driver errors
2. Verify cable connection with iblinkinfo
3. Ensure both ports show "ACTIVE" state in ibstat
4. Confirm subnet manager is running on one host (opensm)

For optimal verbs performance:

• Disable IPoIB if only using verbs: sudo ip link set ib0 down
• Set hugepages: echo 1024 > /proc/sys/vm/nr_hugepages
• Use ibv_devinfo -v to check supported operations

Yes, you can directly connect two Mellanox ConnectX-2 Host Channel Adapters (HCAs) without an InfiniBand switch using a simple passive copper cable (QSA or QSFP). Unlike Ethernet, InfiniBand doesn't require crossover cables - the standard DAC (Direct Attach Copper) cables work fine for point-to-point connections.

# Verify HCA detection
lspci | grep Mellanox
# Expected output should show your ConnectX-2 devices

For RDMA verbs programming, you'll need to configure IPoIB (IP over InfiniBand) first to establish basic communication:

# Load required modules
modprobe mlx4_core
modprobe mlx4_ib
modprobe ib_umad
modprobe ib_uverbs
modprobe rdma_cm
modprobe rdma_ucm

# Bring up IPoIB interface
ip link set mlx4_ib0 up
ip addr add 192.168.1.1/24 dev mlx4_ib0  # On first machine
ip addr add 192.168.1.2/24 dev mlx4_ib0  # On second machine

Here's a simple RDMA ping-pong example using ibv_verbs:

#include 

int main() {
    struct ibv_device **dev_list;
    struct ibv_context *context;
    
    // Get device list
    dev_list = ibv_get_device_list(NULL);
    if (!dev_list) {
        perror("Failed to get IB devices list");
        return 1;
    }

    // Open device context
    context = ibv_open_device(dev_list[0]);
    if (!context) {
        fprintf(stderr, "Couldn't open device\n");
        return 1;
    }

    // Query device attributes
    struct ibv_device_attr device_attr;
    if (ibv_query_device(context, &device_attr)) {
        fprintf(stderr, "Couldn't query device\n");
        return 1;
    }

    printf("Max QP: %d\n", device_attr.max_qp);
    // Continue with QP creation and communication...
    return 0;
}

For optimal performance in direct-connect mode:

• Disable IPoIB when not needed (it adds overhead)
• Set MTU to maximum supported (4KB typically)
• Use hugepages for memory registration
• Enable busyloop polling for latency-sensitive apps

# Set hugepages (run as root)
echo 1024 > /proc/sys/vm/nr_hugepages

# Set MTU
ip link set mlx4_ib0 mtu 4096

If you encounter issues:

1. Verify cable connection (both LEDs should be on)
2. Check dmesg for HCA initialization errors
3. Ensure subnet manager is running (opensm)
4. Verify link state: cat /sys/class/net/mlx4_ib0/carrier

# Start subnet manager
opensm -g 0xfe80000000000000 -B /etc/opensm/opensm.conf