NewsAggregator

App Listed Dormant

A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance

Checked · Sources

Downloads
Not measured
Dependents
Not applicable
Stars
10
Forks
6
Contributors
8
Commits
539

About

From the project README ·

We live in an era where both the demand and quantity of information are enormous. However, the way we store and access that information has remained unchanged, via keywords. Given such a direct association, it is hard to algorithmically determine the relationship between pieces of information.

So we want to introduce a knowledge engine to work with data inference within a given search. Wolfram Alpha does just this, but limited to mathematical context. We will branch beyond this into many subjects and topics including news and knowledge databases.

Thus, we are building a knowledge engine that will utilise machine learning models to show and find information related to a query. Users will be able to search for information that has better association, allowing them the ability to more efficiently learn and expand on a topic

Overview

Setting up the components of News Aggregator is suprisingly straightforward. We rely on Dockerized services in order to ensure ease of use, modularity, reliability and configurability. Each of the Redis, Elasticsearch, Frontend and Backend containers have portions of the repository shared onto this via mounted volumes, including configuration.

Read the full README

From the get go, there isn’t any additional configuration properties to alter, its mostly ensuring the required tooling is installed.

Dependencies

In order to start up the services and work on them there is a few bits a pieces to install. Off the bat you will need to have the following installed:

| Tool | Download | |---------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Git | This should already by installed, but if it isn’t then you can find installation instructions here:
https://git-scm.com/book/en/v2/Getting-Started-Installing-Git | | Docker | Mac: https://docs.docker.com/docker-for-mac/install/
Windows: https://docs.docker.com/docker-for-windows/install/
Linux: https://docs.docker.com/engine/install/ubuntu/ | | Python 3.8 | https://www.python.org/downloads/release/python-3810/ | | Node JS | https://nodejs.org/en/download/ | | An IDE and/or text editor such as:
* VSCode
* IntelliJ
* Sublime Text
* Atom
* PyCharm
* etc | * https://code.visualstudio.com/
* https://www.sublimetext.com/
* https://atom.io/
* https://www.jetbrains.com/idea/
* https://www.jetbrains.com/pycharm/ | | Ansible [Optional] | We use ansible for the deployment system. If you are not working on it then you can skip installing it.
However here is the installation page if need be:
https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html | | Rust + Cargo (Rust CLI) [Optional] | If you are working on the Redis modules, then you’ll need to install Rust and it's Cargo cli tool.
You can do so with the following instructions:
https://www.rust-lang.org/tools/install
https://doc.rust-lang.org/cargo/getting-started/installation.html |

Once you have those installed, we can move on to setting up the repo itself.

Repository Local setup

You’ll first want to clone down the repo onto your local machine by using the following command:

git clone https://github.com/EngineersBox/NewsAggregator.git
Frontend

If you are looking to work on the frontend portion of the repo, then you’ll want to head over to the frontend directory.

  1. Run npm i to install all the relevant npm packges
  2. In a seperate terminal window/tab run npm run serve to start the local development server
  3. Open up your IDE or text editor of choice in the frontend directory
Backend

Running the backend flask API is also straightforward. You’ll need to be in the top level NewsAggregator directory for this.

  1. Run python3 -m pip install -r requirements.txt to setup all the dependencies
  2. Install the SpaCy transform with python3 -m spacy download en_core_web_trf
  3. Ensure you have execute permissions on the start script with sudo chmod +x ./run_server.sh
  4. Run the start script with ./run_server.sh
Dockerized Services

Even when developing for Redis or Elasticsearch, we recommend running them in their Dockerized states. This alleviates having to configure the environment for them to run in.

On the other hand, if you are developing the Dockerized platform, testing deployable situations or otherwise this also applies. Note that the containers will be deployed with the host network attached, allowing you to directly access the ports for services such as Elasticsearch as if they were running on your regular local environment (E.g. localhost:9200)

All of the services can be ran with already define compose files. These can be found in the compose/* directories, depending on the component you are looking for, there will be subdirectories to account for this.

You can start any of these services as containers by running docker compose up -d or docker-compose up -d depending on your version of docker and support.

License
Apache-2.0
Crate facts
Not measured
  • Runs on, Native code and Python bindings read crates on crates.io. This entry lists no crate.

Compare with similar projects

Order: strongest tier first, then most downloads, then name, with unmeasured values last. A library counts downloads over 30 days. An app counts binary downloads per release. An app with none counts its crate downloads over 30 days. ML Rust runs this catalogue, so its closest project always gets a row.

NewsAggregator beside the projects that share a topic with it
ProjectDownloadsDependentsStarsLatest releaseRuns on (GPU)Native codeLicense
NewsAggregator This page App Listed Dormant

A fast, searchable, knowledge engine using various machine learning models to aggregate based on importance, association and relevance

Not measured Not applicable 10 Not measured Not measured Not measured Apache-2.0
whatlang-rs Featured

Fast and lightweight language identification library for Rust.

882,019 downloads over 30 days 74 1,087 0.18.0 GPU support unknown No native code found MIT
whichlang Listed

A blazingly fast and lightweight language detection library for Rust.

52,880 downloads over 30 days 11 464 0.1.1 GPU support unknown No native code found MIT
gline-rs Listed

Inference engine for GLiNER models

40,619 downloads over 30 days 3 173 1.1.0 CUDA, ROCm Binds native code: ort-sys Apache-2.0
rust-bert Listed Dormant

Ready-to-use NLP pipelines and language models

24,605 downloads over 30 days 23 3,091 0.23.0 CUDA Binds native code: torch-sys, ort-sys (feature onnx) Apache-2.0
regexr Listed

A high-performance regex engine built from scratch with JIT compilation and SIMD acceleration

8,809 downloads over 30 days 4 28 0.6.2 GPU support unknown No native code found Apache-2.0 OR MIT
kitoken Listed

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

2,949 downloads over 30 days 3 62 0.11.1 GPU support unknown No native code found BSD-2-Clause
Why ListedNo archive, New or Featured rule places it, so it is Listed.

What it lacks for Featured

What else the check found

  • The last activity is on 12 Oct 2021, 1854 days ago. That reaches the 365 days the rules expect, so the project is dormant. Dormancy rule

Not measured: Latest release, Unpatched vulnerabilities past the grace window, Binary downloads per release, First release.

Exact rules: Listed, Featured app. The check on applied thresholds version 9.

Every rule outcome, as the check records it
  • expected-interval: app: E = 365 days
  • dormancy: last activity 2021-10-12, 1854 days ago, E = 365: dormant
  • featured: not featured: dormant and not stable
  • listed: listed: no archive, new or featured rule applies

Measurements and sources

Each signal row shows the raw value, its source and the fetch date. The Sources list gathers every distinct source on this page. Movement needs more history.

Sources4 sources
AdoptionNot measured
Adoption measurements
SignalValueSourceFetchedCounts
Active dependentsNot measuredYes
New dependents (365 days)Not measuredYes
Peak active dependentsNot measuredYes
Active dependents as a share of the peakNot measuredYes
Binary downloads per releaseNot measuredYes
MaintenanceLatest human commit: 5 years ago8 not measured
Maintenance measurements
SignalValueSourceFetchedCounts
Latest human commit2021-10-12GitHub repositoryYes
Latest releaseNot measuredYes
First releaseNot measuredYes
Latest versionNot measuredYes
Days the oldest outside issue waits for a reply0GitHub repositoryYes
Maintainer replies over 365 days0GitHub repositoryYes
Repository archivedfalseGitHub repositoryYes
Repository gonefalseGitHub repositoryYes
Release commit or manifest in the repoNot measuredYes
Published docsNot measuredYes
Latest release fails cargo checkNot measuredYes
Build check dateNot measuredYes
Build check toolchainNot measuredYes
SecurityNot measured
Security measurements
SignalValueSourceFetchedCounts
Unpatched vulnerabilities past the grace windowNot measuredYes
RustSec unmaintained advisoryNot measuredYes
Deprecation noticeNot measuredYes
Deprecated requirementNot measuredYes
Distribution0 binary downloads over the last 3 releases10 not measured
Distribution measurements
SignalValueSourceFetchedCounts
ReleasesNot measuredYes
Downloads over 30 daysNot measuredYes
Binary downloads over the last 3 releases0GitHub releasesYes
Homebrew installs on request over 365 daysNot measuredYes
Repology repository familiesNot measuredYes
Docker pull growth over 90 daysNot measuredYes
Docker pulls at the growth baselineNot measuredYes
Every version a prereleaseNot measuredYes
Every version 0.0.xNot measuredYes
Every version yankedNot measuredYes
Crate deleted from crates.ioNot measuredYes
Repository10 stars, 6 forks, 8 contributors, 539 commits. Stars alone never qualify a project.1 not measured
Repository measurements
SignalValueSourceFetchedCounts
GitHub stars10GitHub repositoryYes — one part of the repository route
GitHub forks6GitHub repositoryYes — one part of the repository route
Contributors with a GitHub account8GitHub contributorsYes — one part of the repository route
Commits on the default branch539GitHub repositoryNo
Repository's main crate left out of the entryNot measuredYes
Facts that never count
Facts that never count measurements
SignalValueSourceFetchedCounts
LicenseApache-2.0GitHub repositoryNo
Runs on, from listed crates and their direct dependenciesNot measuredNo
What shows each backendNot measuredNo
Native code, from listed crates and their dependenciesNot measuredNo
Python bindings, from direct dependencies of listed cratesNot measuredNo
Crates that depend on a listed crate directlyNot measuredNo

Corrections and review history

Report a correction

No correction applies to this entry.

Judgments

A judgment enters only through an approved issue and a merged pull request.

No judgment applies to this entry.