Using the Nuclia Understanding API

The Nuclia Understanding API (NUA) allows you to process data outside of a Knowledge Box. It is a set of endpoints that allow you to send data to the Progress Agentic RAG platform for processing, or to make calls to models to generate text, summaries, or other outputs. The information is stored temporarily for the duration of the processing, nothing remains in the ProgressAgentic RAG cloud infrastructure.

Get a NUA key

In order to use the Nuclia Understanding API (NUA), you first need to generate an API key.

There are two options to create an API key:

Using the Agentic RAG dashboard top-right menu:
- Click on "NUA keys".
- Click on the "Create new Progress Agentic RAG Understanding API key" button.
- Fill out the form and submit it.
Using the Nuclia API /api/v1/account/{your_account}/nua_client endpoint (see Reference).

Once you have the API key, you can use it in the NUA endpoints by setting the header X-NUCLIA-NUAKEY in each request.

The primary usage of a NUA key is to allow running data processing jobs on the Progress Agentic RAG platform.

It may also be used to create (or delete) Knowledge Boxes via the API – for example, if you need to automate Knowledge Box creation. In that case you will need to select the "Allow management of Knowledge Boxes" checkbox during the NUA key creation.

note

You must obtain your API key at the time it is generated. It cannot be retrieved later.

You can generate a new key, but any ongoing processes associated with the first key will be invalidated.

Processing

Push a file

Processing data works in 2 steps:

First you will upload the file with /processing/upload, which returns an upload token.
Then you will put this token in the processing queue with /processing/push so the file is processed.

Here is a typical example:

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/upload' \
  -X POST \
  -H "X-NUCLIA-NUAKEY: Bearer <your-nua-key>" \
  -H 'content-type: <your-file-type>' \
  -T /path/to/file

Will return something like:

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ1cm46cHJveHkiLCJzdWIiOiJmaWxlIiwiYXVkIjoidXJuOnByb3h5IiwiZXhwIjoxNjU1ODkyMDgwLCJpYXQiOjE2NTU4ODg0ODAsImp0aSI6ImFmN2ZmYjU1ZjAxMjRlMzM5MmRjMDY5MmM2NWI5NzY3Iiwic2l6ZSI6ODk3LCJidWNrZXRfbmFtZSI6ImV1cm9wZS0xLXN0b3JhZ2UtcHJveHkiLCJ1cmkiOiIwZDg2YTFmYTcxN2Q0NDUxYTAwM2Q1NGIyNzM2YTA0ZSIsImRyaXZlciI6MCwiY29udGVudF90eXBlIjoiYXBwbGljYXRpb24vb2N0ZXQtc3RyZWFtIiwiZmlsZW5hbWUiOiIiLCJwYXNzd29yZCI6bnVsbCwibGFuZ3VhZ2UiOm51bGwsInNvdXJjZSI6MCwibWQ1IjpudWxsfQ.VItLa_fUen2Pt5W2440Bjwc7Zx64rv6mJnRMFIdao5o

That is your upload token. Now you can push this token to the processing queue with:

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/push' \
  -X POST \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>' \
  -H 'Content-Type: application/json' \
  --data-raw '{"filefield":{"my_file_1":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ1cm46cHJveHkiLCJzdWIiOiJmaWxlIiwiYXVkIjoidXJuOnByb3h5IiwiZXhwIjoxNjU1ODkxODYyLCJpYXQiOjE2NTU4ODgyNjIsImp0aSI6ImIzMGY4ZTJlNmY3MjQ4OWM5MjJmMTQzNjVjYTdkM2M1Iiwic2l6ZSI6NjM0NywiYnVja2V0X25hbWUiOiJldXJvcGUtMS1zdG9yYWdlLXByb3h5IiwidXJpIjoiMDczNDI5YjRjODM1NDlmMjhjM2U1MzhiOTBhZDFhZTUiLCJkcml2ZXIiOjAsImNvbnRlbnRfdHlwZSI6ImFwcGxpY2F0aW9uL29jdGV0LXN0cmVhbSIsImZpbGVuYW1lIjoiIiwicGFzc3dvcmQiOm51bGwsImxhbmd1YWdlIjpudWxsLCJzb3VyY2UiOjAsIm1kNSI6Ik1qaGlaREpoTkdWaVpHVXhNVGMzTnpFd1lqazRPRE5tT1RjeE1UVXpNR1k9In0.W8wfVj04aYicREYXRatCTb8F8-PD4Ph7MmiTliHi6DI"}}'

Push text

Instead of uploading a file and then pushing it to the queue, you can also push some text directly to the queue.

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/push' \
  -X POST \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>' \
  -H 'Content-Type: application/json' \
  --data '{
    "textfield": {
        "game_theory": {
            "body": "Game theory is the study of mathematical models of strategic interactions among rational agents. It has applications in all fields of social science, as well as in logic, systems science and computer science.",
            "format": 0
        },
        "lorem_ipsum": {
            "body": "<head><title>Lorem Ipsum</title></head><html>Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry'\''s standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book.</html>",
            "format": 1
        },
        "readme.md": {
            "body": "# asyncapi-schema-pydantic\n\nAsyncAPI (v2) specification schema as [Pydantic](https://github.com/samuelcolvin/pydantic) classes.\n\nThe naming of the classes follows the schema in [AsyncAPI specification](https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#schema).\n\n## Installation\n\n`pip install asyncapi-schema-pydantic`",
            "format": 2
        }
    }
  }'

note

You will need to indicate the text format:

PLAIN = 0
HTML = 1
MARKDOWN = 2
RST = 3
JSON = 4
KEEP_MARKDOWN = 5
JSONL = 6
PLAIN_BLANKLINE_SPLIT = 7

Collect the processing outputs

When calling /push, you will get a response containing the processing_id of the processing job.

Processing happens asynchronously, so you will need to regularly check the queue output using one of the two endpoints available to retrieve the results.

Protobuffer results

The /processing/requests/<processing_id> endpoint returns a protobuffer payload with the processing results. You can call it like this:

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/requests/<processing_id>' \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>'

The response is a JSON object containing completed indicating if the processing is complete or not, and in case it is,a response string.

The payload is a base64-encoded protobuffer message. It contains:

the uuid of the processing job
file metadata
extracted text
nested text (like text in an embedded image)
a summary
text blocks (under paragraphs property) and sentences (defined by the position of their first and last characters, plus start time and end time for a video or audio file)
named entities (people, dates, places, organizations, etc.)
links
a thumbnail
embedded files
the vector representations of the text

note

Generated files (thumbnail, extracted embedded files, etc.) are provided as a token. You can download them with the /processing/download endpoint.

Also at any level, if an attribute exceeds a certain size, it will be put in a downloadable file and will be replaced in the document by a file pointer. This will consist of {"file": {"uri": "JWT_TOKEN"}}. The rule is that if the size of the message is greater than 1000000 characters, the biggest parts will be moved to downloadable files. First, the compression process will target vectors. If that is not enough, it will target large field metadata, and finally it will target extracted text.

In Python, you can use nucliadb_protos to decode the results:

import base64
import requests
from nucliadb_protos.writer_pb2 import BrokerMessage

processing_id = "THE_PROCESSING_ID_YOU_RECEIVED_FROM_PUSH"
res = requests.get(f'https://europe-1.nuclia.cloud/api/v1/processing/requests/{processing_id}',headers={
    "X-NUCLIA-NUAKEY": "Bearer YOUR_NUA_KEY",
}).json()

if "payload" in res:
    pb = BrokerMessage()
    pb.ParseFromString(
        base64.b64decode(res["payload"])
    )
    print(pb)
else:
    print('No payload')

For other languages, Nuclia protobuffer models can be found on the NucliaDB GitHub repository.

The payload received from /requests is based on the fdbwriter.BrokerMessage provided by writer.proto.

Streamed results

The /processing/requests/<processing_id>/results endpoint is similar but it returns directly the decoded results as a stream. You can call it like this:

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/requests/<processing_id>/results' \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>'

Moreover the data_types parameter allows you to filter the results by data type. For example:

curl 'https://<zone>.dp.progress.cloud/api/v1/processing/requests/<processing_id>/results?data_types=FullText&data_types=GeneratedFile' \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>'

Use a webhook

If regularly pulling the queue to get the latest process results does not work well in your context, you can also use a webhook URL.

In this case, any time a new result is available Nuclia will post it (i.e. make an actual POST call) to this URL, sending the protobuffer result as payload.

To use a webhook, you will need to fill in the "Webhook URL" field in the NUA key creation form when you create your key.

It cannot be changed afterwards but can be overwritten for a given process by passing webhook_config parameter in /push call:

curl 'https://<zone>.nuclia.cloud/api/v1/processing/push' \
  -X POST \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>' \
  -H 'Content-Type: application/json' \
  --data-raw '{"webhook_config": {"uri": "http://some.where/my-custom-webhook"}, "filefield": {"my_file.pdf":"<your-upload-token>"}}'

webhook_config also accepts a headers parameter where you can set arbitrary headers you may need, for example to authorize your webhook calls.

{
  "webhook_config": {
    "uri": "http://some.where/my-custom-webhook",
    "headers": {
      "api-key": "xxxxxxxxxx"
    }
  }
}

Also, you can choose to use a fixed uri on your token, and provide the headers in the /push calls. In this case you don't need to provide the uri parameter.

note

If an error occurs when posting the result to the webhook URL, the result is stored in the queue and can be retrieved by calling the /pull endpoint (see previous section). The error message is appended to the result.

Model calls

The predict endpoints offer various services:

/predict/chat makes a direct call to an LLM.
/predict/remi calls the REMi model to measure the quality of a RAG response.
/predict/rephrase rephrases a question to make it more suitable for a RAG query.
/predict/rerank reranks a list of results to improve the relevance of the top results.
/predict/sentence computes the embedding of a sentence.
/predict/summarize summarizes a text.

The calls must provides the X-NUCLIA-NUAKEY header with your NUA key:

curl 'https://<zone>.nuclia.cloud/api/v1/predict/chat' \
  -X POST \
  -H 'X-NUCLIA-NUAKEY: Bearer <your-nua-key>' \
  -H 'Content-Type: application/json' \
  --data-raw '{"question": "What is the capital of Senegal?"}'

You can use the Nuclia Python SDK to use these endpoints.

Note: If you have a Knowledge Box and want to use the predict endpoints, they are proxied through the /api/v1/kb/<kbid>/predict endpoint. You will need to provide a KB's regular authentication, a NUA API key will not be necessary.

Get a NUA key​

Processing​

Push a file​

Push text​

Collect the processing outputs​

Protobuffer results​

Streamed results​

Use a webhook​

Model calls​

Get a NUA key

Processing

Push a file

Push text

Collect the processing outputs

Protobuffer results

Streamed results

Use a webhook

Model calls