You can also join our Discord server! If you found Harmony helpful, you can leave us a review!
Read our guide to contributing to Harmony here or read CONTRIBUTING.md.
You can run the walkthrough Python notebook in Google Colab with a single click:
You can also download an R markdown notebook to run in R Studio:
You can run the walkthrough R notebook in Google Colab with a single click: View the PDF documentation of the R package on CRAN
Check out our examples repository at https://github.com/harmonydata/harmony_examples
Harmony is a tool using AI which allows you to compare items from questionnaires and identify similar content. You can try Harmony at https://harmonydata.ac.uk/app and you can read our blog at https://harmonydata.ac.uk/blog/.
You can contact Harmony team at https://harmonydata.ac.uk/, or Thomas Wood at https://fastdatascience.com/.
π₯ Installation instructions (video) π± Looking to try Harmony in the browser?Visit: https://harmonydata.ac.uk/app/
You can also visit our blog at https://harmonydata.ac.uk/
β You need Tika if you want to extract instruments from PDFsDownload and install Java if you don't have it already. Download and install Apache Tika and run it on your computer https://tika.apache.org/download.html
java -jar tika-server-standard-2.3.0.jar
You need a Windows, Linux or Mac system with
You can install from PyPI.
Harmony uses spaCy to help with text extraction from PDFs. spaCy models can be downloaded with the following command in Python:
import harmony
harmony.download_models()
instruments = harmony.example_instruments["CES_D English"], harmony.example_instruments["GAD-7 Portuguese"]
match_response = harmony.match_instruments(instruments)
questions = match_response.questions
similarity = match_response.similarity_with_polarity
harmony.load_instruments_from_local_file("gad-7.pdf")
As an alternative to downloading models, you can set environment variables so that Harmony calls spaCy on a remote server. This is only necessary if you are making a server deployment of Harmony.
HARMONY_DATA_PATH - determines where data files are stored. Defaults to HOME DIRECTORY/harmonyHARMONY_NO_PARSING - set to 1 to import a lightweight variant of Harmony which doesn't support PDF parsing.HARMONY_NO_MATCHING - set to 1 to import a lightweight variant of Harmony which doesn't support matching.You can also create instruments quickly from a list of strings
from harmony import create_instrument_from_list, match_instruments
instrument1 = create_instrument_from_list(["I feel anxious", "I feel nervous"])
instrument2 = create_instrument_from_list(["I feel afraid", "I feel worried"])
match_response = match_instruments([instrument1, instrument2])
If you have a local file, you can load it into a list of Instrument instances:
from harmony import load_instruments_from_local_file
instruments = load_instruments_from_local_file("gad-7.pdf")
Once you have some instruments, you can match them with each other with a call to match_instruments.
from harmony import match_instruments
match_response = match_instruments(instruments)
match_response.questions is a list of the questions passed to Harmony, in order.match_response.similarity_with_polarity is the similarity matrix returned by Harmony.match_response.query_similarity is the degree of similarity of each item to an optional query passed as argument to match_instruments.Harmony defaults to sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (HuggingFace link). However you can use other sentence transformers from HuggingFace by setting the environment HARMONY_SENTENCE_TRANSFORMER_PATH before importing Harmony:
export HARMONY_SENTENCE_TRANSFORMER_PATH=sentence-transformers/distiluse-base-multilingual-cased-v2
Any word vector representation can be used by Harmony. The below example works for OpenAI's text-embedding-ada-002 model as of Apri 2025, provided you have create a paid OpenAI account. However, since LLMs are progressing rapidly, we have chosen not to integrate Harmony directly into the OpenAI client libraries, but instead allow you to pass Harmony any vectorisation function of your choice.
import numpy as np
from harmony import match_instruments_with_function, example_instruments
from openai import OpenAI
client = OpenAI()
model_name = "text-embedding-ada-002"
def convert_texts_to_vector(texts):
vectors = client.embeddings.create(input = texts, model=model_name).data
return np.asarray([vectors[i].embedding for i in range(len(vectors))])
instruments = example_instruments["CES_D English"], example_instruments["GAD-7 Portuguese"]
match_response = match_instruments_with_function(instruments, None, convert_texts_to_vector)
Download and install Docker:
Open a Terminal and run
docker run -p 8000:80 harmonydata/harmonyapi
Then go to http://localhost:8000 in your browser to see the API.
You can now install and run the front end locally: https://www.youtube.com/watch?v=1xp3Uh6dptg
Looking for the Harmony API?Visit: https://github.com/harmonydata/harmonyapi
If you are a Docker user, you can run Harmony from a pre-built Docker image.
If you'd like to contribute to this project, you can contact us at https://harmonydata.ac.uk/ or make a pull request on our Github repository. You can also raise an issue.
Test code is in tests/ folder using unittest.
The testing tool tox is used in the automation with GitHub Actions CI/CD. Since the PDF extraction also needs Java and Tika installed, you cannot run the unit tests without first installing Java and Tika. See above for instructions.
Install tox and run it:
In our configuration, tox runs a check of source distribution using check-manifest (which requires your repo to be git-initialized (git init) and added (git add .) at least), setuptools's check, and unit tests using pytest. You don't need to install check-manifest and pytest though, tox will install them in a separate environment.
The automated tests are run against several Python versions, but on your machine, you might be using only one version of Python, if that is Python 3.9, then run:
Thanks to GitHub Actions' automated process, you don't need to generate distribution files locally.
βοΈContinuous integration/deployment to PyPIThis package is based on the template https://pypi.org/project/example-pypi-package/
This package
master or main branch, and is published when create a releaseThe code to re-release Harmony on PyPI is as follows:
source activate py311
pip install twine
rm -rf dist
python setup.py sdist
twine upload dist/*
Harmony is a collaboration project between Ulster University, University College London, the Universidade Federal de Santa Maria, and Fast Data Science. Harmony has been funded by Wellcome as part of the Wellcome Data Prize in Mental Health and by Economic and Social Research Council (ESRC).
The core team at Harmony is made up of:
Harmony itself is under MIT License. Copyright (c) 2023 Ulster University (https://www.ulster.ac.uk). The third party resources used include:
Licenses of third party software Third party dependency License Use Python BSD-style custom license Programming language - all of Harmony runs based on Python and so this can't be replaced Java Different options available such as Oracle and IBM Programming language used to run Tika, used for PDF parsing. If we replace Tika we may no longer need Java. Sentence Transformers Apache Library for running transformer models Transformers Apache Library for running transformer models Pandas BSD 3-Clause Handling tables inside Harmony - mainly for reading/writing Excels Tika Apache Parsing PDFs into plain text including OCR. Runs in Java LXML BSD Reading the output of Tika's PDF parsing Langdetect Apache Detecting language of text XlsxWriter BSD 2-Clause Writing Excels Openpyxl MIT Writing Excels Numpy custom license which appears to be BSD 3-Clause Dependency of the transformers libraries Scikit-Learn BSD 3-Clause Machine learning models for extracting the questions from PDFs Scikit-Learn CRFSuite MIT Machine learning models for extracting the questions from PDFs Scipy custom license which appears to be BSD 3-Clause Machine learning models for extracting the questions from PDFs Huggingface Hub Apache Connects to HuggingFace Hub, online catalogue of transformer models Third party software only used for the API Third party dependency License Use FastAPI MIT Runs the API Pydantic MIT Ensures that data going in and out of the API is consistently formatted Pydantic Settings MIT Ensures that data going in and out of the API is consistently formatted Uvicorn BSD 3-Clause Runs the API APScheduler MIT Periodically downloads Mental Health Catalogue data and similar - could potentially be removed Third party software only used for using LLMs from cloud providers Third party dependency License Use VertexAI Apache Calls Google Vertex API LLMs OpenAI Apache Calls OpenAI LLMs π How do I cite Harmony?You can cite our validation paper:
McElroy, Wood, Bond, Mulvenna, Shevlin, Ploubidis, Scopel Hoffmann, Moltrecht, Using natural language processing to facilitate the harmonisation of mental health questionnaires: a validation study using real-world data. BMC Psychiatry 24, 530 (2024), https://doi.org/10.1186/s12888-024-05954-2
A BibTeX entry for LaTeX users is
@article{mcelroy2024using,
title={Using natural language processing to facilitate the harmonisation of mental health questionnaires: a validation study using real-world data},
author={McElroy, Eoin and Wood, Thomas and Bond, Raymond and Mulvenna, Maurice and Shevlin, Mark and Ploubidis, George B and Hoffmann, Mauricio Scopel and Moltrecht, Bettina},
journal={BMC psychiatry},
volume={24},
number={1},
pages={530},
year={2024},
publisher={Springer}
}
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4