Troubleshooting

Profiling 

This section covers how to set up and use profiling tools for cgrates to diagnose performance issues. You can enable profiling through configuration, runtime flags, or APIs.

For more information on profiling in Go, see the Go Diagnostics: Profiling documentation.

Configuration 

There are two ways to set up profiling:

Using JSON Configuration 

Enable profiling by adding the pprof_path under the http section in your JSON config file:

{
    "listen": {
        "http": ":2080",
        "http_tls": ":2280"
    },
    "http": {
        "pprof_path": "/debug/pprof/"
    }
}

Profiling is enabled by default and exposes the /debug/pprof/ endpoint. You can access it through the address set in the listen section (http or http_tls). To turn off profiling, set pprof_path to an empty string "".

Using Runtime Flags 

You can also control profiling with runtime flags when starting cgr-engine:

$ cgr-engine -help
Usage of cgr-engine:
 -cpuprof_dir string
       Directory for CPU profiles
 -memprof_dir string
       Directory for memory profiles
 -memprof_interval duration
       Interval between memory profile saves (default 15s)
 -memprof_maxfiles int
       Number of memory profiles to keep (most recent) (default 1)
 -memprof_timestamp
       Add timestamp to memory profile files

Generating Profile Data 

Let’s assume the profiling interface is available at http://localhost:2080/debug/pprof/.

Note

Profiling started with flags or APIs can be stopped using the corresponding API calls. If you start profiling on startup using flags and don’t stop it manually, a profile will be automatically generated when the engine shuts down. The same applies if you start profiling via API and forget to stop it before shutting down the engine.

CPU Profiling 

Here’s how to generate CPU profile data:

Web Browser: Go to http://localhost:2080/debug/pprof/ in your browser. Click “profile” to start a 30-second CPU profile.
Custom Duration: Add the seconds parameter to set a different duration: http://localhost:2080/debug/pprof/profile?seconds=5.

Command Line: Use curl to download the profile:

curl -o cpu.prof http://localhost:2080/debug/pprof/profile?seconds=5

APIs: Use CoreSv1.StartCPUProfiling and CoreSv1.StopCPUProfiling APIs:

{
    "method": "CoreSv1.StartCPUProfiling",
    "params": [{
        "DirPath": "/tmp"
    }],
    "id": 1
}

{
    "method": "CoreSv1.StopCPUProfiling",
    "params": [],
    "id": 1
}

Startup Profiling: Profile the entire runtime by specifying a directory with the -cpuprof_dir flag:
```
cgr-engine -cpuprof_dir=/tmp [other flags]
```

Memory Profiling 

Generate memory profile data like this:

Web Browser: Visit http://localhost:2080/debug/pprof/ to create a memory snapshot. Use ?debug=2 (or ?debug=1) for human-readable output. If the debug parameter is omitted or set to 0, the output will be in binary format.

Command Line: Use curl to download the memory profile:

curl -o mem.prof http://localhost:2080/debug/pprof/heap

Automated Profiling: Use the CoreSv1.StartMemoryProfiling API for periodic memory snapshots:

{
    "method": "CoreSv1.StartMemoryProfiling",
    "params": [{
        "DirPath": "/tmp",
        "Interval": 5000000000,
        "MaxFiles": 5,
        "UseTimestamp": true
    }],
    "id": 1
}

Note

Interval is in nanoseconds. Future updates will allow using time strings (e.g., 5s, 1h) or seconds as an integer.

Other Useful Profiles 

The /debug/pprof/ endpoint offers more useful profiles:

Goroutine Profile (/debug/pprof/goroutine): View or download goroutine stack dumps.
Mutex Profile (/debug/pprof/mutex): Find bottlenecks where goroutines wait for locks.
Block Profile (/debug/pprof/block): Identify where goroutines block waiting on synchronization primitives.
Thread Create Profile (/debug/pprof/threadcreate): Show stack traces that led to the creation of new OS threads.
Execution Trace: For information on generating and analyzing execution traces, see the Tracing section below.

Analyzing Profiles 

The main tool for analyzing profiles is go tool pprof. It helps visualize and analyze profiling data. You can use it with both downloaded profile files and directly with URLs.

Command-Line Analysis 

For CPU profiles:

go tool pprof cpu.prof
# or
go tool pprof http://localhost:2080/debug/pprof/profile

For memory profiles:

go tool pprof mem.prof
# or
go tool pprof http://localhost:2080/debug/pprof/heap

This opens an interactive terminal. Use commands like top, list, web, and svg to explore the profile.

Hint

Run go tool pprof -h for more information on available commands and options.

Visual Analysis 

Create visual representations of your profiling data:

SVG: Generate an SVG graph:

go tool pprof -svg cpu.prof > cpu.svg
# or
go tool pprof -svg mem.prof > mem.svg

Web Interface: Use -http for an interactive visualization in your browser:
```
go tool pprof -http=:8080 cpu.prof
# or
go tool pprof -http=:8080 mem.prof
```
Note

You might need to install the graphviz package.

Tracing 

Execution tracing provides a detailed view of runtime behavior of your Go program.

For detailed information on tracing in Go, see the Go Diagnostics: Execution Tracing documentation.

To generate and analyze trace data:

# Generate trace data
curl -o trace.out http://localhost:2080/debug/pprof/trace?seconds=5

# Analyze trace data
go tool trace trace.out

This opens a browser interface for detailed execution analysis.

Debugging 

This section covers how to set up and use Delve, a Go debugger, with cgrates.

For detailed information on debugging Go programs, see the Go Diagnostics: Debugging documentation.

Installation 

To install Delve, run:

go install github.com/go-delve/delve/cmd/dlv@latest

Basic Usage 

There are several ways to use Delve with cgrates:

Start cgr-engine in debug mode:

dlv exec /path/to/cgr-engine -- --config_path=/etc/cgrates --logger=*stdout

Attach to a running instance:

ENGINE_PID=$(pidof cgr-engine)
dlv attach $ENGINE_PID

Debug tests:

dlv test github.com/cgrates/cgrates/apier/v1 -- -test.run=TestName

Hint

For better debugging, disable optimizations (-N) and inlining (-l) when building cgr-engine:

go install -gcflags="all=-N -l" -ldflags "-X 'github.com/cgrates/cgrates/utils.GitLastLog=$GIT_LAST_LOG'" github.com/cgrates/cgrates/cmd/cgr-engine

Handling Crashes 

To capture more information when cgrates crashes:

Enable core dump generation:

ulimit -c unlimited
GOTRACEBACK=crash cgr-engine -config_path=/etc/cgrates

Analyze core dumps with Delve:
```
dlv core /path/to/cgr-engine core
```

Common Debugging Commands 

Once in a Delve debug session, you can use these common commands:

break or b: Set a breakpoint
continue or c: Run until breakpoint or program termination
next or n: Step over to next line
step or s: Step into function call
print or p: Evaluate an expression
goroutines: List current goroutines
help: Show help for commands

For more information on using Delve, refer to the Delve Documentation.