API

warning

This page is incomplete and may not contain correct information.

Features

The MADI-API project provides the following features:

Authentication and Authorization: The API supports authentication and authorization using the @feathersjs/authentication package. It includes strategies for Google Identity-Aware Proxy (IAP) authentication and user management with role-based access control (RBAC). The API enforces user abilities and permissions defined in the @casl/ability library.
Database Management: The API connects to a PostgreSQL database using the Knex.js query builder library. It includes database migrations and seed files to manage the database schema and populate it with initial data. The database connection configuration is stored in the knexfile.js file.
RESTful API Endpoints: The API exposes RESTful endpoints for various resources, including users, documents, chats, chunks, uploads, and tags. It follows the OpenAPI specification for documenting the endpoints and includes security definitions and response schemas.
Service Architecture: The API is structured using a service-oriented architecture. Each resource has its own service module that handles CRUD operations and business logic. The services follow a common structure and implement hooks for authentication, validation, and error handling.
Data Validations and Schemas: The API includes data validation using the Ajv library and defines schemas and validators for each resource. It ensures that the input data and query parameters conform to the specified schema and enforces data consistency.
Error Handling and Logging: The API includes hooks and middleware for error handling and logging. It logs errors using the Winston library and provides customizable error messages and error codes for different scenarios.
Test Suite: The API includes a test suite for unit testing the endpoints, services, and utility functions. It uses the Mocha test framework, Chai assertion library, and Supertest library for making HTTP requests.
Continuous Integration/Deployment (CI/CD): The API includes a GitHub Actions workflow (deploy-cloudrun.yml) for automated deployment to Google Cloud Run. It builds and deploys the application when changes are pushed to the main branch.
Documentation: The API includes API documentation generated from the OpenAPI specification. It provides detailed information about the available endpoints, their request/response structures, and examples.
Custom Plugins and Tools: The API includes custom plugins and tools, such as a tool for retrieving the current weather using OpenWeather API and a tool for searching scholarly articles using the Semantic Scholar API. These plugins are organized in the src/plugin-tools directory.
Extensibility and Modularity: The API is designed to be extensible and modular. It follows best practices for organizing code and provides clear separation of concerns between different modules and components.
Contribution Guidelines: The API includes contribution guidelines and a code of conduct for contributors. It encourages community participation through pull requests, discussions, and issue reporting.

Repository Structure

└── madi-api/
    ├── .cicd
    │   └── upsertTag.cjs
    ├── .github
    │   └── workflows
    │       └── deploy-cloudrun.yml
    ├── Dockerfile
    ├── config
    │   ├── custom-environment-variables.json
    │   └── default.yml
    ├── docker-compose.yml
    ├── knexfile.js
    ├── migrations
    │   ├── 20231122170107_user.js
    │   ├── 20231122170718_document.js
    │   ├── 20231122171125_chat.js
    │   └── 20231219155838_chunk.js
    ├── package-lock.json
    ├── package.json
    ├── public
    │   ├── index.html
    │   └── spec.json
    ├── seeds
    │   └── dev-admin.js
    ├── specifications
    │   ├── openai.yml
    │   ├── openapi
    │   │   ├── _merged.yml
    │   │   ├── errors.yml
    │   │   ├── security.yml
    │   │   ├── shared.yml
    │   │   └── spec.yml
    │   └── postman
    │       ├── contract.postman_collection.json
    │       └── postman.json
    ├── src
    │   ├── app.js
    │   ├── auth
    │   │   ├── abilites.js
    │   │   ├── authentication.js
    │   │   ├── authorize.hook.js
    │   │   ├── createUser.hook.js
    │   │   ├── googleIAP.strategy.js
    │   │   └── permissions.yml
    │   ├── configuration.js
    │   ├── hooks
    │   │   └── log-error.js
    │   ├── index.js
    │   ├── logger.js
    │   ├── plugin-tools
    │   │   ├── index.js
    │   │   ├── investigations
    │   │   ├── semantic-scholar
    │   │   ├── sources
    │   │   ├── tags
    │   │   └── weather
    │   ├── postgresql.js
    │   ├── services
    │   │   ├── chats
    │   │   ├── chunks
    │   │   ├── documents
    │   │   ├── index.js
    │   │   ├── uploads
    │   │   ├── users
    │   │   └── utils
    │   └── validators.js
    └── test
        ├── app.test.js
        ├── client.test.js
        └── services
            ├── chats
            ├── chunks
            ├── documents
            ├── investigations
            ├── madi
            ├── sources
            ├── tags
            ├── uploads
            └── users

Getting Started

Requirements

Ensure you have the following dependencies installed on your system:

JavaScript: version x.y.z

Installation

Clone the madi-api repository:

git clone https://github.com/nasa-madi/madi-api

Change to the project directory:

cd madi-api

Install the dependencies:

npm install

Running madi-api

Use the following command to run madi-api:

node app.js

Tests

To execute tests, run:

npm test

Project Roadmap

► INSERT-TASK-1
► INSERT-TASK-2
► ...

Connecting to an IAP-Protected API on GCP

To securely connect to an API protected by Google Cloud's Identity-Aware Proxy (IAP), follow these detailed steps to authenticate requests using a service account or user-managed identity. This guide assumes that you have already set up IAP to protect your API and have appropriate permissions configured for accessing the resource.

1. Setting Up Environment Variables

First, define the environment and client ID variables that you will use in your authentication requests. These variables are essential for acquiring an authentication token and for making API requests to the IAP-protected service.

gcloud config set project <PROJECT_ID>
gcloud iap oauth-brands list

This will respond with something like:

name: projects/[PROJECT_NUMBER]/brands/[BRAND_ID]
applicationTitle: [APPLICATION_TITLE]
supportEmail: [SUPPORT_EMAIL]
orgInternalOnly: true

Next, copy the projects and brands link and run the following command

gcloud iap oauth-clients list projects/351312167908/brands/351312167908

You should see a response like the following

name: projects/[PROJECT_NUMBER]/brands/[BRAND_NAME]/identityAwareProxyClients/[CLIENT_ID]
secret: [CLIENT_SECRET]
displayName: [NAME]

Now you have sufficient information to generate a token. Use the [CLIENT_ID] and the right env variables below.

ENV="dev"  #or 'test' or 'prod'
DOMAIN="example.app" # or simliar domain where the app is hosted
CLIENT_ID="XXXXXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.apps.googleusercontent.com"
TOKEN=$(gcloud auth print-identity-token --audiences=$CLIENT_ID)

ENV: This variable represents the deployment environment (e.g., dev, prod). Adjust it according to your specific environment names.
DOMAIN: This variable represents the domain that you will be connecting to through IAP.
CLIENT_ID: This is the client ID associated with the IAP. You can find this ID in your Google Cloud Console under the IAP section.
TOKEN: This command generates an identity token for the specified client ID using the currently authenticated gcloud session. Ensure your gcloud is authenticated with a user or service account that has permission to access the IAP-protected resource.

2. Verify Service Account or User Access

Before making broader API calls, verify that the service account or user has the necessary permissions and can authenticate through IAP by creating a test user. This step confirms that your setup can successfully post data to the IAP-protected API. For more information see instructions here

curl --location --request POST "https://$ENV.$DOMAIN/api/users/" \
--header "Authorization: Bearer $TOKEN" \
--data '{}'

This curl command makes a POST request to the API to create a new user. If the users exists for the provided token it will error and say that you cannot create new users.
It uses the Bearer token obtained in the previous step for authentication.
Ensure that there is no trailing whitespace or additional characters that might break the header formatting or the URL.

3. General Usage of Token

After successfully verifying that the token works for a simple POST request, you can use the same method to authenticate other types of requests to the API. Adjust the HTTP method and endpoint according to the specific actions you need to perform. For example, here's a request that lists the available tools (provided by plugins):

curl --location "https://$ENV.$DOMAIN/api/tools" \
--header "Authorization: Bearer $TOKEN"

Best Practices and Troubleshooting

Token Expiry: Be aware that the tokens generated are temporary. For long-term operations, ensure your application can handle token renewal.
Error Handling: Implement robust error handling in your code to manage and log errors returned by the IAP or the API.
Security Practices: Keep your client IDs and service account credentials secure. Avoid hard-coding sensitive information directly in your application's source code.

This setup ensures that your applications can securely access APIs protected by Google's Identity-Aware Proxy using authenticated tokens that verify the identity and permissions of the requester.

Connecting to the Database Directly

The easiest way to connect to the database is through a bastion host. The commands for this can be fetched from the terraform output with the following command:

cd ../terraform
terraform workspace select <ENV> #develop #test #production
terraform output

You must be in the /terraformm folder of the main /madi project and must have permissions to run the terraform.

Contributing

Contributions are welcome! Here are several ways you can contribute:

Submit Pull Requests: Review open PRs, and submit your own PRs.
Join the Discussions: Share your insights, provide feedback, or ask questions.
Report Issues: Submit bugs found or log feature requests for Madi-api.

Contributing Guidelines

Fork the Repository: Start by forking the project repository to your GitHub account.
Clone Locally: Clone the forked repository to your local machine using a Git client.
```
git clone https://github.com/nasa-madi/madi-api
```
Create a New Branch: Always work on a new branch, giving it a descriptive name.
```
git checkout -b new-feature-x
```
Make Your Changes: Develop and test your changes locally.
Commit Your Changes: Commit with a clear message describing your updates.
```
git commit -m 'Implemented new feature x.'
```
Push to GitHub: Push the changes to your forked repository.
```
git push origin new-feature-x
```
Submit a Pull Request: Create a PR against the original project repository. Clearly describe the changes and their motivations.

Once your PR is reviewed and approved, it will be merged into the main branch.

License

This project is protected under the MIT License. For more details, refer to the LICENSE file.

Acknowledgments

Special thanks go out to the MADI contributors:

Return

API

Features​

Repository Structure​

Getting Started​

Installation​

Running madi-api​

Tests​

Project Roadmap​

Connecting to an IAP-Protected API on GCP​

1. Setting Up Environment Variables​

2. Verify Service Account or User Access​

3. General Usage of Token​

Best Practices and Troubleshooting​

Connecting to the Database Directly​

Contributing​

License​

Acknowledgments​