NewsAggregator
App Listed Dormant
A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance
- Downloads
- Not measured
- Dependents
- Not applicable
- Stars
- 10
- Forks
- 6
- Contributors
- 8
- Commits
- 539
About
We live in an era where both the demand and quantity of information are enormous. However, the way we store and access that information has remained unchanged, via keywords. Given such a direct association, it is hard to algorithmically determine the relationship between pieces of information.
So we want to introduce a knowledge engine to work with data inference within a given search. Wolfram Alpha does just this, but limited to mathematical context. We will branch beyond this into many subjects and topics including news and knowledge databases.
Thus, we are building a knowledge engine that will utilise machine learning models to show and find information related to a query. Users will be able to search for information that has better association, allowing them the ability to more efficiently learn and expand on a topic
Overview
Setting up the components of News Aggregator is suprisingly straightforward. We rely on Dockerized services in order to ensure ease of use, modularity, reliability and configurability. Each of the Redis, Elasticsearch, Frontend and Backend containers have portions of the repository shared onto this via mounted volumes, including configuration.
Read the full README
From the get go, there isn’t any additional configuration properties to alter, its mostly ensuring the required tooling is installed.
Dependencies
In order to start up the services and work on them there is a few bits a pieces to install. Off the bat you will need to have the following installed:
| Tool | Download |
|---------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Git | This should already by installed, but if it isn’t then you can find installation instructions here:
https://git-scm.com/book/en/v2/Getting-Started-Installing-Git |
| Docker | Mac: https://docs.docker.com/docker-for-mac/install/
Windows: https://docs.docker.com/docker-for-windows/install/
Linux: https://docs.docker.com/engine/install/ubuntu/ |
| Python 3.8 | https://www.python.org/downloads/release/python-3810/ |
| Node JS | https://nodejs.org/en/download/ |
| An IDE and/or text editor such as:
* VSCode
* IntelliJ
* Sublime Text
* Atom
* PyCharm
* etc | * https://code.visualstudio.com/
* https://www.sublimetext.com/
* https://atom.io/
* https://www.jetbrains.com/idea/
* https://www.jetbrains.com/pycharm/ |
| Ansible [Optional] | We use ansible for the deployment system. If you are not working on it then you can skip installing it.
However here is the installation page if need be:
https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html |
| Rust + Cargo (Rust CLI) [Optional] | If you are working on the Redis modules, then you’ll need to install Rust and it's Cargo cli tool.
You can do so with the following instructions:
https://www.rust-lang.org/tools/install
https://doc.rust-lang.org/cargo/getting-started/installation.html |
Once you have those installed, we can move on to setting up the repo itself.
Repository Local setup
You’ll first want to clone down the repo onto your local machine by using the following command:
git clone https://github.com/EngineersBox/NewsAggregator.git
Frontend
If you are looking to work on the frontend portion of the repo, then you’ll want to head over to the frontend directory.
- Run
npm ito install all the relevant npm packges - In a seperate terminal window/tab run
npm run serveto start the local development server - Open up your IDE or text editor of choice in the
frontenddirectory
Backend
Running the backend flask API is also straightforward. You’ll need to be in the top level NewsAggregator directory for this.
- Run
python3 -m pip install -r requirements.txtto setup all the dependencies - Install the SpaCy transform with
python3 -m spacy download en_core_web_trf - Ensure you have execute permissions on the start script with
sudo chmod +x ./run_server.sh - Run the start script with
./run_server.sh
Dockerized Services
Even when developing for Redis or Elasticsearch, we recommend running them in their Dockerized states. This alleviates having to configure the environment for them to run in.
On the other hand, if you are developing the Dockerized platform, testing deployable situations or otherwise this also applies. Note that the containers will be deployed with the host network attached, allowing you to directly access the ports for services such as Elasticsearch as if they were running on your regular local environment (E.g. localhost:9200)
All of the services can be ran with already define compose files. These can be found in the compose/* directories, depending on the component you are looking for, there will be subdirectories to account for this.
You can start any of these services as containers by running docker compose up -d or docker-compose up -d depending on your version of docker and support.
Compatibility
- License
- Apache-2.0
- Crate facts
- Not measured
- Runs on, Native code and Python bindings read crates on crates.io. This entry lists no crate.
Compare with similar projects
Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.
| Project | Downloads | Dependents | Stars | Latest release | Runs on (GPU) | Native code | License |
|---|---|---|---|---|---|---|---|
|
A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance |
Not measured | Not applicable | 10 | Not measured | Not measured | Not measured | Apache-2.0 |
|
whatlang-rs Featured
Fast and lightweight language identification library for Rust. |
882,019 downloads over 30 days | 74 | 1,087 | 0.18.0 | GPU support unknown | No native code found | MIT |
|
whichlang Listed
A blazingly fast and lightweight language detection library for Rust. |
52,880 downloads over 30 days | 11 | 464 | 0.1.1 | GPU support unknown | No native code found | MIT |
|
gline-rs Listed
Inference engine for GLiNER models |
40,619 downloads over 30 days | 3 | 173 | 1.1.0 | CUDA, ROCm | Binds native code: ort-sys | Apache-2.0 |
|
Ready-to-use NLP pipelines and language models |
24,605 downloads over 30 days | 23 | 3,091 | 0.23.0 | CUDA | Binds native code: torch-sys, ort-sys (feature onnx) | Apache-2.0 |
|
regexr Listed
A high-performance regex engine built from scratch with JIT compilation and SIMD acceleration |
8,809 downloads over 30 days | 4 | 28 | 0.6.2 | GPU support unknown | No native code found | Apache-2.0 OR MIT |
|
kitoken Listed
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization |
2,949 downloads over 30 days | 3 | 62 | 0.11.1 | GPU support unknown | No native code found | BSD-2-Clause |
Why ListedNo archive, New or Featured rule places it, so it is Listed.
What it lacks for Featured
- The project is dormant and not finished. Dormancy rule
What else the check found
- The last activity is on 12 Oct 2021, 1854 days ago. That reaches the 365 days the rules expect, so the project is dormant. Dormancy rule
Not measured: Latest release, Unpatched vulnerabilities past the grace window, Binary downloads per release, First release.
Exact rules: Listed, Featured app. The check on applied thresholds version 9.
Every rule outcome, as the check records it
expected-interval: app: E = 365 daysdormancy: last activity 2021-10-12, 1854 days ago, E = 365: dormantfeatured: not featured: dormant and not stablelisted: listed: no archive, new or featured rule applies
Measurements and sources
Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.
Sources4 sources
- Description: the GitHub repository description
- GitHub repository, fetched
- GitHub releases, fetched
- GitHub contributors, fetched
AdoptionNot measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Active dependents | Not measured | Yes | ||
| New dependents (365 days) | Not measured | Yes | ||
| Peak active dependents | Not measured | Yes | ||
| Active dependents as a share of the peak | Not measured | Yes | ||
| Binary downloads per release | Not measured | Yes |
MaintenanceLatest human commit: 5 years ago8 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Latest human commit | 2021-10-12 | GitHub repository | Yes | |
| Latest release | Not measured | Yes | ||
| First release | Not measured | Yes | ||
| Latest version | Not measured | Yes | ||
| Days the oldest outside issue waits for a reply | 0 | GitHub repository | Yes | |
| Maintainer replies over 365 days | 0 | GitHub repository | Yes | |
| Repository archived | false | GitHub repository | Yes | |
| Repository gone | false | GitHub repository | Yes | |
| Release commit or manifest in the repo | Not measured | Yes | ||
| Published docs | Not measured | Yes | ||
| Latest release fails cargo check | Not measured | Yes | ||
| Build check date | Not measured | Yes | ||
| Build check toolchain | Not measured | Yes |
SecurityNot measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Unpatched vulnerabilities past the grace window | Not measured | Yes | ||
| RustSec unmaintained advisory | Not measured | Yes | ||
| Deprecation notice | Not measured | Yes | ||
| Deprecated requirement | Not measured | Yes |
Distribution0 binary downloads over the last 3 releases10 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| Releases | Not measured | Yes | ||
| Downloads over 30 days | Not measured | Yes | ||
| Binary downloads over the last 3 releases | 0 | GitHub releases | Yes | |
| Homebrew installs on request over 365 days | Not measured | Yes | ||
| Repology repository families | Not measured | Yes | ||
| Docker pull growth over 90 days | Not measured | Yes | ||
| Docker pulls at the growth baseline | Not measured | Yes | ||
| Every version a prerelease | Not measured | Yes | ||
| Every version 0.0.x | Not measured | Yes | ||
| Every version yanked | Not measured | Yes | ||
| Crate deleted from crates.io | Not measured | Yes |
Repository10 stars, 6 forks, 8 contributors, 539 commits. Stars alone never qualify a project.1 not measured
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| GitHub stars | 10 | GitHub repository | Yes — one part of the repository route | |
| GitHub forks | 6 | GitHub repository | Yes — one part of the repository route | |
| Contributors with a GitHub account | 8 | GitHub contributors | Yes — one part of the repository route | |
| Commits on the default branch | 539 | GitHub repository | No | |
| Repository's main crate left out of the entry | Not measured | Yes |
Facts that never count
| Signal | Value | Source | Fetched | Counts |
|---|---|---|---|---|
| License | Apache-2.0 | GitHub repository | No | |
| Runs on, from listed crates and their direct dependencies | Not measured | No | ||
| What shows each backend | Not measured | No | ||
| Native code, from listed crates and their dependencies | Not measured | No | ||
| Python bindings, from direct dependencies of listed crates | Not measured | No | ||
| Crates that depend on a listed crate directly | Not measured | No |
Corrections and review history
No correction applies to this entry.
Judgments
A judgment enters only through an approved issue and a merged pull request.
No judgment applies to this entry.