| Good code :white_check_mark: | Bad code :x: |
| ```python def display_last_uni_swaps( top: int = 10, sortby: str = "timestamp", descend: bool = False, export: str = "",) -> None: ``` | ```python def display_last_uni_swaps( top: int, sortby: str, descend: bool, export: str,) -> None: ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python def get_coins( top: int = 250, category: str = "") -> pd.DataFrame: ``` | ```python def load( file: str, file_types: list, data_files: Dict[Any, Any], data_examples: Dict[Any, Any],) -> pd.DataFrame: ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python data: pd.Series, dataset_name: str, y_label: str, ``` | ```python data: pd.Series, dataset: str, column: str, ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python def get_gaintopain_ratio(portfolio: PortfolioEngine) -> pd.DataFrame: """...""" gtp_period_df = portfolio_helper.get_gaintopain_ratio( portfolio.historical_trade_data, portfolio.benchmark_trades, portfolio.benchmark_returns) return gtp_period_df ``` | ```python def get_gaintopain_ratio(self) -> pd.DataFrame: """...""" vals = list() for period in portfolio_helper.PERIODS: port_rets = portfolio_helper.filter_df_by_period(self.portfolio_returns, period) bench_rets = portfolio_helper.filter_df_by_period(self.benchmark_returns, period) ... ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python # [fred_view.py] def display_yieldcurve(country: str): df = fred_model.get_yieldcurve(country) β¦ # [fred_model.py] def get_yieldcurve(country: str) -> pd.Dataframe: β¦ ``` | ```python # [fred_view.py] def display_bondscrv(country: str): df = fred_model.get_yieldcurve(country) β¦ # [fred_model.py] def get_yldcurve(country: str) -> pd.Dataframe: β¦ ``` |
It is important to keep a coherent UI/UX throughout the terminal. These are the rules we must abide:
- There is 1 single empty line between user input and start of the command output.
- There is 1 single empty line between command output and the user input.
- The menu help has 1 empty line above text and 1 empty line below. Both still within the rectangular panel.
- From menu help rectangular panel there's no empty line below - this makes it more clear to the user that they are inside such menu.
## External API Keys
### Creating API key
OpenBB Terminal currently has over 100 different data sources. Most of these require an API key that allows access to some free tier features from the data provider, but also paid ones.
When a new API data source is added to the platform, it must be added through [credentials_model.py](/openbb_terminal/core/models/credentials_model.py).
In order to do that, you'll simply need to choose from one the following files:
1. [local_credentials.json](openbb_terminal/miscellaneous/models/local_credentials.json) --> credentials that should only be stored locally and not pushed to the [OpenBB Hub](https://my.openbb.co/) like brokerage keys or other very sensitive or personal to the user.
2. [hub_credentials.json](openbb_terminal/miscellaneous/models/hub_credentials.json) --> credentials that should be stored in the [OpenBB Hub](https://my.openbb.co/) like API keys to access your favorite providers.
Then just update [all_api_keys.json](openbb_terminal/miscellaneous/models/all_api_keys.json) with the instructions to get
the api key from the data source website. Make sure that this file has the correct `.json` format, otherwise the API keys page in the Hub will break (e.g. in json the last element key-value pair shouldn't be followed by a comma, and the last object in a list of dictionaries should also not be followed by a comma).
> Note: By differentiating between local and hub credentials, we can ensure that the user's credentials are not pushed to the [OpenBB Hub](https://my.openbb.co/) and are only stored locally. This does not mean that the credentials are not secure in the OpenBB Hub, but rather that the user can choose to store them locally if they wish.
### Setting and checking API key
One of the first steps once adding a new data source that requires an API key is to add that key to our [keys_controller.py](/openbb_terminal/keys_controller.py). This menu allows the user to set API keys and check their validity.
The following code allows to check the validity of the Polygon API key.
```python
def check_polygon_key(show_output: bool = False) -> str:
"""Check Polygon key
Parameters
----------
show_output: bool
Display status string or not. By default, False.
Returns
-------
str
Status of key set
"""
current_user = get_current_user()
if current_user.credentials.API_POLYGON_KEY == "REPLACE_ME":
logger.info("Polygon key not defined")
status = KeyStatus.NOT_DEFINED
else:
r = request(
"https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/day/2020-06-01/2020-06-17"
f"?apiKey={current_user.credentials.API_POLYGON_KEY}"
)
if r.status_code in [403, 401]:
logger.warning("Polygon key defined, test failed")
status = KeyStatus.DEFINED_TEST_FAILED
elif r.status_code == 200:
logger.info("Polygon key defined, test passed")
status = KeyStatus.DEFINED_TEST_PASSED
else:
logger.warning("Polygon key defined, test inconclusive")
status = KeyStatus.DEFINED_TEST_INCONCLUSIVE
if show_output:
console.print(status.colorize())
return str(status)
```
Note that there are usually 3 states:
- **defined, test passed**: The user has set their API key and it is valid.
- **defined, test failed**: The user has set their API key but it is not valid.
- **not defined**: The user has not defined any API key.
Note: Sometimes the user may have the correct API key but still not have access to a feature from that data source, and that may be because such feature required an API key of a higher level.
A function can then be created with the following format to allow the user to change its environment key directly from the terminal.
```python
def call_polygon(self, other_args: List[str]):
"""Process polygon command"""
parser = argparse.ArgumentParser(
add_help=False,
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
prog="polygon",
description="Set Polygon API key.",
)
parser.add_argument(
"-k",
"--key",
type=str,
dest="key",
help="key",
)
if not other_args:
console.print("For your API Key, visit: https://polygon.io")
return
if other_args and "-" not in other_args[0][0]:
other_args.insert(0, "-k")
ns_parser = self.parse_simple_args(parser, other_args)
if ns_parser:
self.status_dict["polygon"] = keys_model.set_polygon_key(
key=ns_parser.key, persist=True, show_output=True
)
```
# ADVANCED
## Important functions and classes
### Base controller class
This `BaseController` class is inherited by all controllers on the terminal.
This class contains both important variables and methods that are common across all terminal controllers.
**CHOICES_COMMON**: List of common commands across all controllers
- `cls`: clear screen
- `home`: go back to the main root
- `about`: allows to open our documentation directly on the menu or command
- `h`, `?` and `help`: display the help menu the user is in
- `q`, `quit` and `..`: go back to one menu above
- `exit`: exit the platform
- `r` and `reset`: reset the platform (reading code and settings again but going into the same state)
- `support`: create a support request ticket
All of these variables have a `call_FUNCTION` associated with them.
Worthy methods to mention are:
- `load_class`: Checks for an existing instance of the controller before creating a new one to speed up access to that menu.
- `custom_reset`: Should be used by controllers that rely on a state variable - meant to be overridden. They should add the commands necessary to have the same data loaded.
- `print_help`: Meant to be overridden by each controller
- `parse_input`: Processes the string the user inputs into a list of actionable commands
- `switch`: Acts upon the command action received
- `parse_known_args_and_warn`: Parses the command with the `-` and `--` flags and variables. Some built-in flags are:
- `export_allowed`: Which can be set to `_NO_EXPORT_`, `_EXPORT_ONLY_RAW_DATA_ALLOWED_`, `_EXPORT_ONLY_FIGURES_ALLOWED_` and `_EXPORT_BOTH_RAW_DATA_AND_FIGURES_`
- `raw`: Displaying the data raw
- `limit`: Number of rows to display
- `menu`: Most important method. When a menu is executed, the way to call it is through `stocks_menu.menu()`
## Default Data Sources
The document [openbb_default.json](openbb_terminal/miscellaneous/sources/openbb_default.json) contains all data sources that the terminal has access to and specifies the data source utilized by default for each command.
The convention is as follows:
```python
{
"stocks": {
"search": [
"FinanceDatabase"
],
"quote": [
"FinancialModelingPrep"
],
"tob": [
"CBOE"
],
"candle": [],
"codes": [
"Polygon"
],
"news": [
"Feedparser",
"NewsApi",
],
...
```
The way to interpret this file is by following the path to a data source, e.g.
- `stocks/search` relies on `FinanceDatabase`
- `stocks/candle` does not rely on any data source. This means that it relies on data that has been loaded before.
- `stocks/load` relies on `YahooFinance`, `AlphaVantage`, `Polygon` or `EODHD`.
- **The order is important as the first data source is the one utilized by default.**
- `stocks/options/unu` relies on `FDScanner`.
- `stocks/options/exp` relies on `YahooFinance` by default but `Tradier` and `Nasdaq` sources are allowed.
> Note: The default data sources can be changed directly in the [OpenBB Hub](https://my.openbb.co/) by the user and automatically synchronized with the terminal on login.
## Export Data
In the `_view.py` files it is common having at the end of each function `export_data` being called. This typically looks like:
```python
export_data(
export,
os.path.dirname(os.path.abspath(__file__)),
"pt",
df_analyst_data,
sheet_name,
fig,
)
```
Let's go into each of these arguments:
- `export` corresponds to the type of file we are exporting.
- If the user doesn't have anything selected, then this function doesn't do anything.
- The user can export multiple files and even name the files.
- The allowed type of files `json,csv,xlsx` for raw data and `jpg,pdf,png,svg` for figures depends on the `export_allowed` variable defined in `parse_known_args_and_warn`.
- `os.path.dirname(os.path.abspath(__file__))` corresponds to the directory path
- This is important when `export folder` selected is the default because the data gets stored based on where it is called.
- If this is called from a `common` folder, we can use `os.path.dirname(os.path.abspath(__file__)).replace("common", "stocks")` instead
- `"pt"` corresponds to the name of the exported file (+ unique datetime) if the user doesn't provide one
- `df_analyst_data` corresponds to the dataframe with data.
- `sheet_name` corresponds to the name of the sheet in the excel file.
- `fig` corresponds to the figure to be exported as an image or pdf.
If `export_allowed=EXPORT_BOTH_RAW_DATA_AND_FIGURES` in `parse_known_args_and_warn`, valid examples are:
- `cmd --export csv`
- `cmd --export csv,png,jpg`
- `cmd --export mydata.csv`
- `cmd --export mydata.txt,alsomydata.csv,alsoalsomydata.png`
Note that these files are saved on a location based on the environment variable: `EXPORT_FOLDER_PATH`. Which can be set in `settings/export`.
The default location is the `exports` folder and the data will be stored with the same organization of the terminal. But, if the user specifies the name of the file, then that will be dropped onto the folder as is with the datetime attached.
## Queue and pipeline
The variable `self.queue` contains a list of all actions to be run on the platform. That is the reason why this variable is always passed as an argument to a new controller class and received back.
```python
self.queue = self.load_class(
DarkPoolShortsController, self.ticker, self.start, self.stock, self.queue
)
```
Example:
If a user is in the root of the terminal and runs:
```shell
stocks/load AAPL/dps/psi -l 90
```
The queue created becomes:
`self.queue = ["stocks", "load AAPL", "dps", "psi -l 90"]`
And the user goes into the `stocks` menu and runs `load AAPL`. Then the queue is updated to
`self.queue = ["dps", "psi -l 90"]`
At that point the user goes into the `dps` menu and runs the command `psi` with the argument `-l 90` therefore displaying price vs short interest of the past 90 days.
## Auto Completer
In order to help users with a powerful autocomplete, we have implemented our own (which can be found [here](/openbb_terminal/custom_prompt_toolkit.py)).
The queue, discussed in the previous section [Queue and pipeline](#queue-and-pipeline), is expected to link together with the autocompletion in order to provide the user with the available options for each command.
Here is an example of how it will look like:
```bash
2023 Apr 11, 11:41 (π¦) /stocks/dps/ $ psi
--nyse
--help
-h
--export
--raw
--limit
-l
--source
```
> Where `nyse`, `help`, `h`, `export`, `raw`, `limit`, `l` and `source` are the available options for the `psi` command.
> Those are selectable using the arrow keys and the `tab` key.
The list of options for each command is automatically generated, if you're interested take a look at its implementation [here](/openbb_terminal/core/completer/choices.py).
To leverage this functionality, you need to add the following line to the top of the desired controller:
```python
CHOICES_GENERATION = True
```
Here's an example of how to use it, on the [`forex` controller](/openbb_terminal/forex/forex_controller.py):
```python
class ForexController(BaseController):
"""Forex Controller class."""
CHOICES_COMMANDS = [
"fwd",
"candle",
"load",
"quote",
]
CHOICES_MENUS = [
"forecast",
"qa",
"ta",
]
RESOLUTION = ["i", "d", "w", "m"]
PATH = "/forex/"
FILE_PATH = os.path.join(os.path.dirname(__file__), "README.md")
CHOICES_GENERATION = True
def __init__(self, queue: Optional[List[str]] = None):
"""Construct Data."""
super().__init__(queue)
self.fx_pair = ""
self.from_symbol = ""
self.to_symbol = ""
self.source = get_ordered_list_sources(f"{self.PATH}load")[0]
self.data = pd.DataFrame()
if session and get_current_user().preferences.USE_PROMPT_TOOLKIT:
choices: dict = self.choices_default
choices["load"].update({c: {} for c in FX_TICKERS})
self.completer = NestedCompleter.from_nested_dict(choices)
...
```
In case the user is interested in a **DYNAMIC** list of options which changes based on user's state, then a class method must be defined.
The example below shows an excerpt from `update_runtime_choices` method in the [`options` controller](/openbb_terminal/stocks/options/options_controller.py).
```python
def update_runtime_choices(self):
"""Update runtime choices"""
if session and get_current_user().preferences.USE_PROMPT_TOOLKIT:
if not self.chain.empty:
strike = set(self.chain["strike"])
self.choices["hist"]["--strike"] = {str(c): {} for c in strike}
self.choices["grhist"]["-s"] = "--strike"
self.choices["grhist"]["--strike"] = {str(c): {} for c in strike}
self.choices["grhist"]["-s"] = "--strike"
self.choices["binom"]["--strike"] = {str(c): {} for c in strike}
self.choices["binom"]["-s"] = "--strike"
```
This method should only be called when the user's state changes leads to the auto-complete not being accurate.
In this case, this method is called as soon as the user successfully loads a new ticker since the options expiry dates vary based on the ticker. Note that the completer is recreated from it.
## Logging
A logging system is used to help tracking errors inside the OpenBBTerminal.
This is storing every logged message inside the following location :
`$HOME/OpenBBUserData/logs`
Where $HOME is the user home directory, for instance:
- `C:\Users\foo` if you are in Windows and your name is foo
- `/home/bar/` if you are in macOS or Linux and your name is bar
The user can override this location using the settings key `OPENBB_USER_DATA_DIRECTORY`.
If you want to log a particular message inside a function you can do like so:
```python
import logging
logger = logging.getLogger(__name__)
def your_function() -> pd.DataFrame:
logger.info("Some log message with the level INFO")
logger.warning("Some log message with the level WARNING")
logger.fatal("Some log message with the level FATAL")
```
You can also use the decorator `@log_start_end` to automatically record a message every time a function starts and ends, like this:
```python
import logging
from openbb_terminal.decorators import log_start_end
logger = logging.getLogger(__name__)
@log_start_end(log=logger)
def your_function() -> pd.DataFrame:
pass
```
> **Note**: if you don't want your logs to be collected, you can set the `OPENBB_LOG_COLLECT` environment variable on your `.env` file to `False`.
>
> **Disclaimer**: all the user paths, names, IPs, credentials and other sensitive information are anonymized, [take a look at how we do it](/openbb_terminal/core/log/generation/formatter_with_exceptions.py).
## Internationalization
WORK IN PROGRESS - The menu can be internationalised BUT we do not support yet help commands`-h` internationalization.
In order to add support for a new language, the best approach is to:
1. Copy-paste `i18n/en.yml`
2. Rename that file to a short version of language you are translating to, e.g. `i18n/pt.yml` for portuguese
3. Then just update the text on the right. E.g.
```text
stocks/NEWS: latest news of the company
```
becomes
```text
stocks/NEWS: mais recentes notΓcias da empresa
```
Note: To speed up translation, the team developed a [script](/i18n/help_translation.ipynb) that uses Google translator API to help translating the entire `en.yml` document to the language of choice. Then the output still needs to be reviewed, but this can be a useful bootstrap.
This is the convention in use for creating a new key/value pair:
- `stocks/search` - Under `stocks` context, short command `search` description on the `help menu`
- `stocks/SEARCH` - Under `stocks` context, long command `search` description, when `search -h`
- `stocks/SEARCH_query` - Under `stocks` context, `query` description when inquiring about `search` command with `search -h`
- `stocks/_ticker` - Under `stocks` context, `_ticker` is used as a key of a parameter, and the displayed parameter description is given as value
- `crypto/dd/_tokenomics_` - Under `crypto` context and under `dd` menu, `_tokenomics_` is used as a key of an additional information, and the displayed information is given as value
## Settings
The majority of the settings used in the OpenBB Terminal are handled using [Pydantic Dataclasses](https://docs.pydantic.dev/usage/dataclasses/).
Some examples are:
1. [SystemModel](openbb_terminal/core/models/system_model.py)
2. [UserModel](openbb_terminal/core/models/user_model.py)
3. [CredentialsModel](openbb_terminal/core/models/credentials_model.py)
4. ...
This means that the settings are pretty much validated and documented automatically, as well as centralized in a single place.
This allows us to develop faster, efficiantly and with predictability.
Depending on your use case you'll most likely need to interact with these dataclasses or expand them with new settings.
**Disclaimer**: avoid at all costs to use the `os.environ` or `os.getenv` methods to retrieve settings. Settings should be retrieved using the appropriate methods from the respective class.
Here is an example of **accessing** a setting:
```python
from openbb_terminal.core.session.current_system import get_current_system
system = get_current_system()
system = get_current_system()
print(system.VERSION)
# 3.0.0
```
And here is an example of **changing** a setting:
```python
from openbb_terminal.core.session.current_system import get_current_system
set_system_variable("TEST_MODE", True)
```
## Write Code and Commit
At this stage it is assumed that you have already forked the project and are ready to start working.
### Pre Commit Hooks
Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every
commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements.
By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change
while not wasting time with trivial style nitpicks.
Install the pre-commit hooks by running: `pre-commit install`.
### Coding
Although the Coding Guidelines section has been already explained, it is worth mentioning that if you want to be faster
at developing a new feature, you may implement it first on a `jupyter notebook` and then carry it across to the
terminal. This is mostly useful when the feature relies on scraping data from a website, or implementing a Neural
Network model.
### Git Process
1. Create your Feature Branch, e.g. `git checkout -b feature/AmazingFeature`
2. Check the files you have touched using `git status`
3. Stage the files you want to commit, e.g.
`git add openbb_terminal/stocks/stocks_controller.py openbb_terminal/stocks/stocks_helper.py`.
Note: **DON'T** add `config_terminal.py` or `.env` files with personal information, or even `feature_flags.py` which is user-dependent.
4. Write a concise commit message under 50 characters, e.g. `git commit -m "meaningful commit message"`. If your PR
solves an issue raised by a user, you may specify such issue by adding #ISSUE_NUMBER to the commit message, so that
these get linked. Note: If you installed pre-commit hooks and one of the formatters re-formats your code, you'll need
to go back to step 3 to add these.
### Branch Naming Conventions
The accepted branch naming conventions are:
- `feature/feature-name`
- `hotfix/hotfix-name`
- `release/2.1.0` or `release/2.1.0rc0`.
- `bugfix/bugfix-name`
- `docs/docs-name`
All `feature/feature-name` and `bugfix/bugfix-name` related branches can only have PRs pointing to `develop` branch. `release/*` branches can only have PRs pointing to `main` branch, while `hotfix/hotfix-name` should first be merged to `main` and then into `develop` to sync the hotfix changes.
When `develop` branch is merged to `main`, a GitHub action will run scripts that generate documentation content (reference sections, data models, etc.) and trigger the website deployment. Those scripts can be found in the following path `website/generate_*.py`.
The `develop` branch is only merged to `main` right before a new release, but sometimes you might need to update the website in-between releases. To do this follow these steps:
1. create `docs/[my-update]` branch from `main`
2. commit your changes to `docs/[my-update]`
3. merge `docs/[my-update]` into `main` -> website deployment triggered
4. merge `docs/[my-update]` into `develop` -> NO website deployment, just to sync branches
## Installers
When implementing a new feature or fixing something within the codebase, it is necessary to ensure that it is working
appropriately on the terminal. However, it is equally as important to ensure that new features or fixes work on the
installer terminal too. This is because a large portion of users utilize the installer to use OpenBB Terminal.
More information on how to build an installer can be found [here](build/README.md).