Beancount advancement Thread

Bojack · December 6, 2019, 12:46pm

You sound a lot like Hedgehog (he doesn’t seem to visit anymore ). He also said: use only open source software, and run all the rest on a VM. I’m too lazy and convenient for this, but I see the point…

Now, I’ve been working on putting my transactions into Beancount. And I have 3 questions.

What aggregation style do you recommend: Assets:AssetType:Institution or Assets:Institutution:AssetType? (Assets:Cash:UBS or Assets:UBS:Cash) The example.beancount uses the 2nd style. But in the 1st style you can aggregate by asset type, regardless of institution. What makes more sense in the long term?
Coming back to shared expenses. If I spend the whole month on groceries, restaurant, trips (various categories) for me and my friend, and at the end of the month he pays me back a lump sum, and I don’t know anymore exactly for which categories does he pay back, then which is more sensible: to record it as a) Income:Friend -1’000 CHF or b) Expenses:Repayment -1’000 CHF? Option A seems logical, but with option B (recording a negative expense) you can correct overblown expenses so they reflect the reality more.
How should I record currency exchange? Let’s say I send CHF to Revolut, exchange to EUR and send back to my bank. When I try to make a transaction with -1’000 CHF and + 900 EUR, he complains that the transaction does not add to 0…

tarioch · December 6, 2019, 1:39pm

I’m actually running more and more of my stuff in docker whenever I can.
For cloud stuff you then build up on the docker images and go with kubernetes to have a full app running. (that’s how I actually run my fava setup)

Tino · December 6, 2019, 1:48pm

Could you please develop a bit your setup? I’m very interested. Some examples of your configuration with docker-compose…etc… would be nice.

Which version of beancount/fava’s docker are you using? I use docker a lot but I found several versions in git and I don’t know if you have some advice.
Thank you.

Bojack · December 6, 2019, 2:02pm

For anybody who stumbles upon it, I figured it out…

2017-04-21 * "TransferWise" "Convert CHF to EUR"
  Assets:Cash:TransferWise                         -1005.00 CHF @@ 934.89 EUR
  Assets:Cash:TransferWise                           934.89 EUR

Could you share a noob’s guide on how to setup fava in the cloud? With login & password, of course. Do you need to pay or are there free hosting options available?

tarioch · December 6, 2019, 2:13pm

My main fava I’m running on a personal kubernetes system I have setup with kubeadm.
I created a helm chart for easy installation of fava onto kubernetes https://github.com/tarioch/helm
Right now I did not yet push my Dockerfile to a public location, it’s another thing I plan to do.

My beancount setup consists of the following things

beancount: normally the latest release on pypi
fava: for some editing, reporting and running importers, also pypi release
importers: whenever I can I have an importer to load transactions in
smart_importer: auto classify transactions, with my data set it’s now about 90% correct, so on each import I just have to adjust a handful of movements by hand.
prices: On a daily cron I fetch prices and fx rates. Most of it using free alphavantage api.
Custom reporting ui: I have my personal reporting ui built on top of beancount that shows me some data that I want (such as rate of return for my assets)

Bojack · December 6, 2019, 2:18pm

Would you be so kind and lay out a step by step noob guide? I have no idea what Kubernetes is, and barely understand Docker, but then again, no idea about dockerfile.

And as said, is there a possibility to use 3rd party hosting and not self host? And can there be authentication implemented, so that my whole transaction history is not for everybody to see?

tarioch · December 6, 2019, 2:21pm

As for a newbs guide. I would be very hesitant to run my finance stuff in the cloud unless you know what you are doing.

Bojack · December 6, 2019, 2:24pm

Sure, I would not put account numbers etc. But yeah, maybe you’re right. On the other hand, if you host it somewhere with HTTPS authentication, then what different is it to Google Drive / Dropbox?

Btw, I found a nice list of Beancount relevant links:

tarioch · December 6, 2019, 2:31pm

All the cloud hosting providers nowadays provide kubernetes services as well but it requires some stuff to learn.
If you go with a cloud provider then you need to learn a bit about helm

https://helm.sh/

And then you can deploy a lot of apps using it. The good thing with it is that you it standardizes a lot of things. E.g. once you understand how to expose such an app with something called an ingress where you can put protection on it works the same way for all apps.

The difference between that and something like dropbox is that here you are running the app, so you need to make sure it’s secured and updated if there’s new security holes.

Maybe one thing to note is that I think beancount is mostly targeting developers. Text files with source control is great for devs. Having your own importers and the ability to tweak things is great. If you are not a dev or interested in this, something like gnucash will probably fit much better.

Bojack · December 6, 2019, 2:38pm

I’m sorry if I sound a bit ungrateful, but I was hoping for a noob’s guide, and now you’re introducing some Helm thing, which manages some Kubernetes thing, which manages some Docker thing . Imagine you’re explaining this to your mom .

Not if your app is sealed of in some HTTPS authentication method. When you open the website, the browser asks for login / pass and the authentication is managed by the protocol, not by the software (fava). That’s the way I imagine it. And they discussed it here:

github.com/beancount/fava

Make fava password-protected

opened 06:57PM - 02 Jun 16 UTC

closed 03:37PM - 13 Jun 16 UTC

aumayr

discussion feature deployment

Research/discuss if there is a way to enable some sort of authentication for the… fava webserver and if this even makes sense. As a responsive layout is underway it may be nice to let fava run "public" and access it via the internet (Browser, smartphone, etc.). This needs at least password-protection (or some sort of authentication) and maybe encryption (HTTPS). Does this make sense?

tarioch · December 6, 2019, 2:46pm

Yeah sorry about that. I think for a newbie you best forget about kubernetes. The easiest is probably to just run it with docker locally on your machine.
That way you just have to do two things.

Install docker (should be quite straight forward)

Run a docker image built by someone else (I’ll get that done if no one else has it already done)

I’ll see about providing the details for running it.

This does not solve the things like managing your beancount files (I would recommend a version control system like git, or at least some way to back them up). And you still have to look into importers for your specific banks.

Bojack · December 6, 2019, 3:10pm

Actually, in the Github issue I linked there is a discussion about it. And it links to yet another issue:

github.com/beancount/fava

Add Docker images

opened 03:36PM - 13 Jun 16 UTC

closed 06:37AM - 01 Oct 16 UTC

aumayr

feature deployment

(This is the successor to #314) @yegle has made a Docker image for fava: https:…//hub.docker.com/r/yegle/fava/ @xentac already provided example-files how to use OAuth with nginx: > https://github.com/xentac/beancount-oauth2 > > Here is a skeleton config showing what I had to do. It doesn't support the url limiting. To do that, you need to change the nginx config like they say here: https://github.com/bitly/oauth2_proxy/#configuring-for-use-with-the-nginx-auth_request-directive I propose the following: - Add two Dockerfiles/images to the `contrib/docker`-folder: One with the latest stable version of fava, and one to build fava from source - Add configuration (via env-variables) to enable/disable OAuth Unfortunately, I do not have any experience with Docker, so any contributions are welcome. @yegle Can we use your images? Are you interested in this?

So there is this guy Yegle that hosts his image:

https://hub.docker.com/r/yegle/fava/

And then aumayr (the creator of fava) made this:

https://hub.docker.com/u/fava

All that being said, what’s the added benefit of me hosting fava locally through Docker? I already installed it and it is running without Docker, so I guess the only reason would be to protect my PC from unauthorised scripts?

Yeah, I got a bitbucket account, and I at least managed to figure GIT out. I definitely intend to backup my beancount file using git.

Ahh, I tell you… I work with software which uses custom IDE (eclipse-based, nothing fancy like VS Code), custom versioning system (not git) and there is never any deployment. So whenever some “true” developers (I say true, because I am practically a programmer, but never deal with all the popular tech) start throwing these modern technologies at me, like git, docker, openshift and whatnot, I get a serious anxiety attack.

tarioch · December 6, 2019, 5:01pm

So here’s the detailed instructions.

It looks like currently only the image from yegle exists. As far as I can tell this contains fava and beancount but not additional things like smart_importer, that’s then something for another day.

What you have to do is the following.

Assuming you have your beancount files in /home/foobar/bean and the main beancount file is called main.beancount

docker run --rm -it -v /home/foobar/bean:/bean -p 5000:5000 yegle/
fava /bean/main.beancount

and after a couple of seconds you should be able to go to http://localhost:5000 and see fava. If you stop the docker run command it should automatically shut everything down again.

To explain what this does

docker run
Command to run an image

–rm -it
Remove the docker run once you’re done and keep attached to the image (so you see output of fava on the command line and can stop everything once you’re done)

-v /home/foobar/bean:/bean
Mount your directory /home/foobar/bean into the directory /bean inside the container

-p 5000:5000

Mount your local port 5000 to the port 5000 in the container, if port 5000 is already used locally you can change the first number to something else and then browse to that port

yegle/fava
name of the image, will automatically be downloaded the first time you execute this

/bean/main.beancount
the name and location of your main beancount file inside the container

tarioch · December 6, 2019, 5:04pm

If you have everything already running there’s not much benefit. The main benefit is if you don’t have it running you don’t need to deal with python and getting everything installed. The other thing is that if you have other apps it keeps things isolated and they don’t conflict with each other. E.g beancount might use python 3.5 and then you have another app which uses python 3.8 and you might run into conflicts.

nugget · December 6, 2019, 6:05pm

my way of profiting from docker is (hopefully, not yet done) i can have my bencount file in my cloud and play with it on any system where i can have docker running.

Tino · December 7, 2019, 4:07pm

Hi Tarioch,

I’ve been checking a bit your git project.

Even if I consider myself a “pro” sometimes I’m lost…specially when I see in the code: “Class”
Well, is a semi-joke, I’m not a real programmer but I’m an advance user.

Could you please explain how to compile, use, install your scripts? I’m specially interested in the Alphavantage script… Honestly, I don’t know how to use. I tried to execute the .py directly but doesn’t work ;-(

Thank you in advance.

Tino.

tarioch · December 7, 2019, 4:24pm

I’m still working on getting this code organized and on pypi which should make it easier to install. Right now you can try to download/clone the whole repository and then do

pip (or pip3) install .

afterwards you should be able to use the alphavantage price source as a a standard price source in beancount. There’s some more documentation available here

The basic is you specify the info for fetching prices (symbol and which plugin to use) on the comodity. So something like

2010-01-01 commodity SLICHA
price: “CHF:plugins.prices.alphavantage/SLICHA.SW”

For the alphavantage plugin to work you will need to request a personal api key from their site

https://www.alphavantage.co/

and define the environment variable ALPHAVANTAGE_API_KEY with it.

You can then use the bean-price cli for fetching prices (part of standard beancount). What I do with it is then store this output into a beancount file to not having to refetch them all the time.

Tino · December 7, 2019, 6:48pm

Thank you!

I’ll try. I’ve been using bean-price for a while with Yahoo, so I have everything running except the driver.
I use Alphavantage for another program and I already have the API_KEY.

Regards.

tarioch · December 7, 2019, 11:38pm

I have now a very early version out on pypi.

pip install tariochbctools

Everything is in the tariochbctools namespace. E.g. for pricing it should be

2010-01-01 commodity SLICHA
price: “CHF: tariochbctools.plugins.prices.alphavantage/SLICHA.SW”

Bojack · December 8, 2019, 5:29pm

I have another question: has anyone cracked how to transfer securities between accounts? I bought some BTC at coinbase, and I don’t want to treat them as currency, so I recorded the cost price. Then I moved the BTC to my ledger stick. Whatever I try, beancount does not seem to do what I want.

Edit: this seems to have worked:

2018-01-25 * "" "Buy BTC"
  Assets:Coinbase:Cash                             -2000.00 EUR
  Assets:Coinbase:BTC                                  0.22 BTC {{1981.00 EUR}}
  Expenses:Fees                                       19.00 EUR

2018-02-01 * "" "Transfer BTC"
  Assets:Coinbase:BTC                                 -0.22 BTC {{1981.00 EUR}}
  Assets:Ledger:BTC                                    0.22 BTC {{1981.00 EUR}}