Node Operators Guide

This guide covers the detailed operational aspects of running an ENI node, including configuration management, maintenance procedures, and best practices for stable and efficient operation.

Configuration Management

Directory Structure

The configuration for an ENI node is stored in the ~/.eni/config/ directory:

~/.eni/config/
├── app.toml          # Application configuration (gas fees, API settings, pruning, etc.)
├── client.toml       # CLI and client-related settings
├── config.toml       # Core Tendermint settings (network, consensus, and RPC)
├── genesis.json      # Genesis file for the chain, defining the initial state
├── node_key.json     # Unique node identity key for peer-to-peer (P2P) networking
└── priv_validator_key.json  # Validator private signing key (if running as a validator)

Key Configuration Parameters

Network Settings (config.toml)

[p2p]
# Public IP for other nodes to reach you
external_address = "your-public-ip:26656"
# Local address to listen for incoming P2P connections
laddr = "tcp://0.0.0.0:26656"
# Allowed number of peers
max_num_inbound_peers = 40
max_num_outbound_peers = 20
# Network bandwidth limits to prevent congestion
send_rate = 20480000  # 20MB/s
recv_rate = 20480000  # 20MB/s

[rpc]
# RPC listening address
laddr = "tcp://0.0.0.0:26657"
# Maximum concurrent connections
max_open_connections = 900
# Transaction confirmation timeout
timeout_broadcast_tx_commit = "10s"

Application Settings (app.toml)

# Minimum gas prices [to prevent spam transactions]
minimum-gas-prices = "0.01ueni"

[api]
# Enable API server
enable = true
max-open-connections = 1000

[state-commit]
# Use ENI-DB for improved performance
sc-enable = true

[state-store]
# Enable state store for historical queries
ss-enable = true
# Keep 100,000 blocks for querying
# 0 = "keep all"
ss-keep-recent = 100000

Database Management

Database Types

ENI supports two database backends:

1. ENI-DB (Recommended)

• Optimized for performance and sync time

• Reduced resource usage

• Suitable for all nodes

1. Traditional IAVL DB

• Standard Cosmos SDK database

• More extensively tested

ENI-DB Configuration

[state-commit]
sc-enable = true
sc-async-commit-buffer = 100
sc-keep-recent = 1  # Keep only the latest state for performance
sc-snapshot-interval = 10000  # State snapshot every 10,000 blocks

[state-store]
ss-enable = true
ss-backend = "pebbledb"  # Default, required
ss-async-write-buffer = 100
ss-keep-recent = 100000  # Keep the most recent 100,000 blocks
ss-prune-interval = 600  # Pruning cleanup interval

Setting a very small [more frequent] pruning interval may cause conflicts with automatic snapshots. A very large [less frequent] pruning interval means longer total pruning time, which could lead to missed blocks and extended resync times.

Database Maintenance

The database is generally stable and can run without intervention, but occasional attention may be required:

# Check database size
du -sh ~/.eni/data/

# Confirm the correct file paths based on your setup before proceeding
# Before performing database cleanup, ensure you back up critical files:
# - `node_key.json`: Preserves node identity for maintaining peer connections.
# - `priv_validator_key.json`: Validator critical; losing this risks double-signing.
# - `config.toml` and `app.toml`: Retain node-specific configuration.
# - `genesis.json`: Required for proper chain initialization.

# Compact the database to optimize storage (only if needed)
find ~/.eni/data/ -mindepth 1 ! -name 'priv_validator_state.json' -delete && rm -rf ~/.eni/wasm

# Back up the database
cp -r ~/.eni/data/ ~/eni-backup-$(date +%Y%m%d)/

Service Management

Systemd Commands

# Check service status
systemctl status enid

# Start service
systemctl start enid

# Stop service
systemctl stop enid

# Restart service
systemctl restart enid

# View logs in real-time
journalctl -fu enid -o cat

Log Management

Enable log rotation to prevent logs from consuming excessive disk space:

sudo tee /etc/logrotate.d/eni > /dev/null << EOF
/var/log/eni/*.log {
    daily
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 eni eni
    sharedscripts
    postrotate
        systemctl reload enid
    endscript
}
EOF

Update Procedures

Minor Updates

For minor updates that do not break consensus:

# Stop the node
sudo systemctl stop enid

# Update the binary
cd eni-chain
git fetch --all
git checkout [new-version]
make install

# Restart the node
sudo systemctl restart enid

Major Updates

For major upgrades introducing state-breaking changes:

1. Wait for the specified upgrade block height [visible in the upgrade proposal’s 'plan']

2. The node will stop automatically.

3. Update/replace the binary

4. Restart the node.

# After the node stops
cd eni-chain
git pull
git checkout [new-version]
make install
sudo systemctl restart enid

Tip - Build the upgrade before the halt height so it can be quickly swapped in, minimizing downtime.

Performance Optimization

The effectiveness of performance optimizations may vary depending on system hardware, workload, and network conditions. Research and test any changes in a controlled environment before implementation to ensure compatibility with your specific configuration and requirements. Always back up critical data before making modifications.

Memory Management (sysctl Tuning)

Optimizing memory management settings can improve performance and stability, especially for high-load nodes. These settings control swap usage and the handling of dirty pages in RAM.

vm.swappiness = 1  # Reduce swapping to disk, ensuring RAM is used effectively before relying on slower swap space
vm.dirty_background_ratio = 3  # Percentage of system memory that can be filled with "dirty" pages before background writing starts
vm.dirty_ratio = 10  # Maximum percentage of system memory that can be filled with "dirty" pages before triggering a full flush
vm.dirty_expire_centisecs = 300  # Time before dirty data is written to disk (in hundredths of a second)
vm.dirty_writeback_centisecs = 100  # Frequency at which "dirty" pages are written to disk (in hundredths of a second)

Network Stack Optimization

Tuning the network stack can improve packet handling efficiency and throughput, particularly for nodes handling many peers and high transaction volumes.

net.core.somaxconn = 32768  # Maximum number of queued connections
net.core.netdev_max_backlog = 32768  # Number of queued packets allowed before dropping incoming packets
net.ipv4.tcp_max_syn_backlog = 16384  # Number of outstanding SYN requests (half-open connections) allowed before dropping new ones
net.core.rmem_max = 16777216  # Receive buffer size for network sockets
net.core.wmem_max = 16777216  # Send buffer size for network sockets

Storage Optimization

Optimizing storage settings can significantly reduce write latency and improve database performance, especially for nodes using NVMe SSDs.

echo "none" > /sys/block/nvme0n1/queue/scheduler  # Disable I/O scheduler to reduce latency
blockdev --setra 4096 /dev/nvme0n1  # Read-ahead value to optimize sequential reads

Backup and Recovery

Regular Backups

Automate backups to prevent data loss:

#!/bin/bash
BACKUP_DIR="/backup/eni"
DATE=$(date +%Y%m%d)

# Stop the service
systemctl stop enid

# Create the backup
tar czf $BACKUP_DIR/eni-backup-$DATE.tar.gz ~/.eni/

# Restart the service
systemctl start enid

Recovery Procedure

Restore from a backup in case of corruption or accidental deletion:

# Stop the node
systemctl stop enid

# Remove corrupted data
rm -rf ~/.eni/data/

# Restore from backup
tar xzf eni-backup-[date].tar.gz -C /

# Restart the node
systemctl start enid

Troubleshooting

Common Issues and Solutions

1. Synchronization Issues

• Check available disk space (df -h)

• Ensure proper peer connections (enid net info)

• Verify firewall settings (port 26656 open)

1. Performance Issues

• Monitor system resources (htop or iotop)

• Check disk I/O performance (iostat)

• Analyze network traffic (iftop)

1. Database Issues

• Run a database integrity check with:

enid debug dump-db | grep -i error

If errors are detected, consider restoring from a recent backup.

• Consider pruning excessive historical data by adjusting ss-keep-recent in app.toml, or run:

enid unsafe-reset-all --home=$HOME/.eni --keep-addr-book

Alternatively, manually delete old state snapshots to free up space:

rm -rf ~/.eni/data/snapshots/*

Diagnostic Commands

# Check sync status
enid status | jq .SyncInfo

# View peer count
enid net info | jq '.n_peers'

# Check validator status
enid query staking validator $(enid keys show -a $VALIDATOR_KEY)

# View recent blocks
enid query blocks recent

Security Considerations

• Use a firewall and rate limiting to prevent attacks

• Keep the system and node software updated

• Secure SSH access with key-based authentication

• Protect validator keys with offline storage or a hardware security module (HSM)

For more in-depth system and configuration guidance, refer to the [Advanced Configuration and Monitoring Guide](./advanced-config-monitoring).