Environment Setup

LiveRamp Sidecar is designed to run as a container that is co-resident to the exchange application it's serving. However, this is not required. Many Sidecar users run a discrete cluster (often using Kubernetes) that exchange servers can reach over the network.

Runtime Flag Overview

You can begin by running docker run --cap-add IPC_LOCK liveramp/idl-mapper:latest -help to see available options.

Runtime Flags

Usage of /gobin/idl-mapperd:
  -auth-token string
    	Flag to add optional authorization to map endpoint.
  -certificate string
    	Path-to-file with a client certificate and certificate key To use for authentication
  -dry-run
    	Switch for connecting to Check-in service, if 'true' dummy keys will be used
  -error-threshold int
    	Threshold for log error (default 90)
  -help
    	Print this help text and exit
  -http-compression
    	Flag for HTTP compression.
  -id-source string
    	ID source to be set as default (default "liveramp.com")
  -listen string
    	Port or Unix socket to listen to (default "unix:///mnt/sock/idl-mapper-socket")
  -proxy-https-cert string
    	Path to HTTPS proxy verifying certificate
  -proxy-url string
    	HTTP proxy URL
  -separate-passthrough
    	Flag to put Passthrough mappings under a separate source domain
  -show-verification-type
    	Switch to show/hide Verification Type in map response
  -shutdown-interval int
    	Number of hours after which mapper will shutdown if it is unauthorized. If 0, shutdown option is disabled. (default 24)
  -warning-threshold int
    	Threshold for log warning (default 50)

🚧

IPC_LOCK is Required

Note that the Docker commands below include the --cap-add IPC_LOCK flag. Enabling this non-default feature is required to protect the sensitive encryption keys used in the Sidecar appliance.

Client Certificate

The client certificate and client key will be provided by LiveRamp. Running in development mode (by passing the dry-run argument) will skip certificate validation, but will still check for the existence of the flag as a runtime argument.

To obtain a production certificate, you will need to provide LiveRamp with a Pretty Good Privacy (PGP) or GNU Privacy Guard (GPG) public key for secure transmission of your certificate. If you do not already have a GPG key, follow the instructions here. However, you can skip any GitHub-specific steps. You will need to provide LiveRamp with the public key, obtained through thegpg --armor --export <KEY_ID> or similar command.

In the following example, a directory at ~Documents/certs is mounted in the container at /mnt/certs.

dev run example

docker run -v ~/Documents/certs:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " " -dry-run true

Connecting to Sidecar via TCP or Socket

Sidecar exposes either a socket or a network interface using TCP to process API requests. While accessing Sidecar over a Unix socket has a small performance improvement over the TCP connection, when Sidecar is co-resident with the exchange application, use TCP for its relative ease-of-use. All examples shown use port 5000. You can reconfigure as necessary. In production, Sidecar is run with a -listen flag, that accepts addresses of the form unix:///path/to/socket or tcp://:port-number.

TCP Network Interface Example

docker run -v ~/Documents/certs:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate "/mnt/certs/CERTIFICATE_NAME.pem"

Note that the Unix socket path should always fall under the /mnt/sock directory on the container, which can be shared with other containers or mounted externally via a docker volume mount at runtime.

Unix Socket Example

docker run -v ~/Documents/certs:/mnt/certs --cap-add IPC_LOCK -v /path/to/host/socket/dir:/mnt/sock liveramp/idl-mapper:idl-mapper:latest -listen unix:///mnt/sock/idl-mapper-socket -certificate "/mnt/certs/CERTIFICATE_NAME.pem"

In the case above, /path/to/host/socket/dir/idl-mapper-socket will appear on the host.

📘

A note on socket sharing

Docker currently doesn't support sharing sockets over a hypervisor. If you're testing Sidecar on a non-Linux machine and try to mount Sidecar's socket to your host, the socket itself won't appear on the host without an error. Testing must be performed either on a VM or in a native Linux environment.

IDSource

Beginning with the Sidecar v2 release, envelopes may contain more than one source of identifiers. In the initial v2 release, identifiers from LiveRamp (IDSource of liveramp.com) as well as The Trade Desk UID2 IDs (IDSource of uidapi.com) are included. Specifying the IDSource as a runtime flag will select the ID used in the single-id /map endpoint. By default, liveramp.com IDs are selected, but this can be set to a different default by passing the appropriate IDSource value. See API endpoints for more information about working with multiple IDSources, including multiple IDs in a single /map request.

Separate Passthrough

You may wish to send envelopes to downstream platforms in the event that they run their own sidecar instances. In this case, you can either skip Sidecar processing or request that LiveRamp map seats as "passthrough".

By default, passthrough seats will be organized under the liveramp.com IDSource:

JSON

[
  {
    "source": "liveramp.com",
    "mapping": {
      "Envelope Passthrough Seat": "Apgt2tWKDEAsfNlOn8_DTpgojfVeGGhr0mbXul_rx4kAj8_Tkxfk3U3Rwdk0fEQ09c16KuWr3F0kwxvyoIL0tLVPfKZO3kVHfFfWtoaDWVz81r2-cEVSn5Hc21eUPyHZNijGW7HdID6gSXozFVwG2PIDy1DYs7xzpdowOTZol8cNfCw512vlo0Cn9ibPIJ0j1Jdf5oU8rszMxKsMHx8a6Bp1NB1rV7B2s_qx9tNlxjxP7V1OqMI9hJnRIisbOx42-paBMrZmMtMs7YAjVxPlKqEFZBmGlAdeqpEQlwNfpCxcwJ5zCjsHjK9IAB9ga7zszBMvh6VfcaRfmEpnZTmbgCZqAKkKNbR6UzDP8yp2bB1JGhN_MmhAhE0AnIHclBhdqK6a5AHqC-TdEz-pMsUNBDPhFXkchoGcRvm_E_Xef2Ydtu88Xy0byRAOtT1hsCAcujM24RtFrbzX9T4",
      "Test Seat 1": "XY9999Ky296WYD8-BuzIlZabukhPeVGr9s5V80Q4b_LRYWu0g"
    }
  },
  {
    "source": "uidapi.com",
    "mapping": {
      "Test Seat 1": "AgAAABUBOEOjShB4rXDrTQT27nejDDTCuSGcXBt6DbdE+HwxMjOLF5BcYbl02PBDSgxvzOuuM86EV84WmhUWI7x7wIpLyfkK5ZGMBQCVzvc+lhtKXgN8fvQIXrvI+x/KFwBiMHOMAa6axWECnJ6SRP65Ih0RBtAdrx9xoPkT1/A1QjzXTw==",
    }
  }
]

By adding -separate-passthrough to your runtime flags, you can treat it as a discrete IDSource:

Passthrough Flag Enabled

[
  ...
  },
  {
    "source": "passthrough",
    "mapping": {
      "Envelope Passthrough Seat": "Apgt2tWKDEAsfNlOn8_DTpgojfVeGGhr0mbXul_rx4kAj8_Tkxfk3U3Rwdk0fEQ09c16KuWr3F0kwxvyoIL0tLVPfKZO3kVHfFfWtoaDWVz81r2"
    }
  }
]

HTTP Compression

HTTP compression can be used with the -http-compression flag with a value of true. In the request, -H "Accept-Encoding: gzip" or -H "Accept-Encoding: deflate" must be sent within the curl to get a valid response. Compression currently only functions with HTTP/1.1 and does not work if using HTTP/2.

Compression Example

docker run  -v /cert_folder/:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " " -http-compression=true

Compression Request

curl -H "Accept-Encoding: gzip" "localhost:5000/v2/map?env=<envelope>"

> Host: 127....
> User-Agent: curl/7.85.0
> Accept: */*
> Accept-Encoding: gzip

Compression Response

< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 1521
< Content-Encoding: gzip

📘

Testing compression

Adding --output response.gz to the end of curl command will get the response placed in a response.gz file.

Security Measures

The application runs completely memory-locked within the container, in order to prevent any sensitive keys from being written to disk, so the application must be given the permissions and resources to effectively lock enough memory. The application shouldn’t need more than 10 MB, although alongside the image operating system, we recommend allocating ~300 MB of RAM.

The application will not run unless the container has its core file size hard limit set to 0, in order to prevent any sensitive keys from being written out in core dumps. If the container is run with this value set to anything above 0, a warning message will be printed to STDOUT, and the value will be overwritten to 0.

❗️

Important

Attempting to modify directories other than /mnt/sock and /mnt/secrets on the Sidecar can result in triggering tamper-prevention mechanisms, which could result in an account-wide disruption of service.

API Authentication

The API key is used for an extra layer of authentication. To use an API key, contact [email protected] as it requires enablement.

📘

This key will become a requirement in a future release of Sidecar to provide a higher level of security.

There are two ways to use an environment variable as an API key value:

1. Add the environment variable IDL_MAPPER_API_KEY and run the idl-mapper with --env IDL_MAPPER_API_KEY flag by adding it after the docker run command.

Add API Key to environment variable

export IDL_MAPPER_API_KEY=<api_key>

To confirm the API Key is applied to the environmental variable, use: echo $IDL_MAPPER_API_KEY

Run with API Key from Variable

docker run --env IDL_MAPPER_API_KEY -v /cert_folder/:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " "

2. Create a file that contains 'IDL_MAPPER_API_KEY=variable_value'. Run the idl-mapper with the --env-file file_name flag by adding it after the docker run command.

Text File Sample

IDL_MAPPER_API_KEY=<api_key>

Run with API Key from File

docker run --env-file file_name -v /cert_folder/:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " "

Error Messages

Error Message

level=warning msg="Missing API key" err="API key could not be pulled from IDL_MAPPER_API_KEY environment variable"
level=warning msg="check in unsuccessful" err="check in service denied authorization"

When the "Missing API Key" message appears, verify that you have properly added the variable to the request to run Sidecar.

Network Traffic and Telemetry

The Sidecar container must communicate with LiveRamp's services at sidecar.liveramp.com to periodically (every one to five minutes) check in, obtain and renew keys, and to send telemetry and crash reports.

No personal data is sent to LiveRamp, only aggregated counts and information about the Docker runtime. In most cases, connectivity requires no special Docker setup. However, it may be impacted if you're using a special networking mode.

📘

Sidecar IP Addresses

Sidecar connects to sidecar.liveramp.com. To allowlist specific IP addresses, use the following:

• 35.193.49.161
• 35.188.138.165
• 35.202.174.138
• 136.111.154.45

See API endpoints for information on what aggregated telemetry is collected.

HTTP(S) Proxy Support

You can configure Sidecar to use a proxy when calling the check-in service API. This update includes both HTTP and HTTPS support. HTTPS requires a new proxy verifying certificate. To enable proxy support, contact your LiveRamp representative.

To support this functionality, two additional configuration arguments are added:

• proxy-url (Optional) Proxy server URL (e.g., -proxy-url "https\://user:[[email protected]](mailto:[email protected]):3129")
• proxy-https-cert Path to the HTTPS proxy verifying certificate, which is mandatory if an HTTPS proxy is used

Sidecar with HTTP Proxy

docker run -v /cert_folder/:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " " -proxy-url "http://192.168.1.100:3128"

Sidecar with HTTPS Proxy

docker run -v /cert_folder/:/mnt/certs --cap-add IPC_LOCK -p 127.0.0.1:5000:5000 liveramp/idl-mapper:latest -listen tcp://:5000 -certificate " " -proxy-url "https://192.168.1.100:3129" -proxy-https-cert "/mnt/certs/proxy_client_verifying_ca.crt"

ARM64 Support

The default Sidecar image is intended for use on x86 systems. However, LiveRamp also offers ARM64 support (select images with the -arm64 suffix).