guile-vm 0.3

[Keisuke Nishida]

I have checked in the current snapshot of my Guile VM to the CVS.
If anyone is interested, please check it out and try it.

Install
-------

% cd guile/guile-vm
% ./autogen.sh; configure && make install
% ln -s module/{language,system} /usr/local/share/guile/site/

Setup
-----

Add the following lines to your ~/.guile:

(if (string=? (car (command-line)) "guile-vm")
(begin
(use-modules (system repl repl))
(start-repl 'ghil)
(quit)))

Example
-------

% gulie-vm
Guile High Intermediate Language (GHIL) interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@user> (@+ 1 2)
$1 = 3
***@user> ,c -c (@+ 1 2) ;; Compile into GLIL
(@asm (0 0 0 0)
(const 1)
(const 2)
(add)
(return))
***@user> ,c -l (@+ 1 2) ;; Compile into loadable code
0 make-int8:0 ;; 0
1 load-program #0
8 return

Bytecode #0:

0 make-int8:1 ;; 1
1 make-int8 2 ;; 2
3 add
4 return

***@user> ,c (@+ 1 2) ;; Compile and disassemble
Disassembly of #<program 0x80ff863>:

args = 0 rest = 0 locals = 0

Bytecode:

0 make-int8:1 ;; 1
1 make-int8 2 ;; 2
3 add
4 return

***@user> (@define fib
(@lambda (n)
(@if (@< n 2) 1 (@+ (fib (@- n 1)) (fib (@- n 2))))))
***@user> ,trace (fib 3)
(#<program 0x80f607c> 3)
| (#<program 0x80f607c> 2)
| | (#<program 0x80f607c> 1)
| | 1
| | (#<program 0x80f607c> 0)
| | 1
| 2
| (#<program 0x80f607c> 1)
| 1
3
***@user> ,trace -a (fib 1)
0xbfffef68 (call 0) #() (#<program 0x80f6813>)
0x80f6813 (make-int8:1) #() ()
0x80f6814 (load-module) #() (1)
0x80f681a (load-symbol) #() (#<env 40345e00> 1)
0x80f681f (link) #() (fib #<env 40345e00> 1)
0x80f6820 (variable-ref) #() ((fib . #<program 0x80f607c>) 1)
0x80f6821 (tail-call 1) #() (#<program 0x80f607c> 1)
(#<program 0x80f607c> 1)
0x80f607c (local-ref:0) #(1) ()
0x80f607d (make-int8 2) #(1) (1)
0x80f607f (lt?) #(1) (2 1)
0x80f6080 (br-if-not 2) #(1) (#t)
0x80f6082 (make-int8:1) #(1) ()
0x80f6083 (return) #(1) (1)
1
0xbfffef6a (halt) #() (1)
***@user> ,time (fib 30)
$2 = 1346269
clock utime stime cutime cstime gctime
3.42 3.40 0.00 0.00 0.00 0.00
***@user>

Status
------

Currently, the Scheme interpreter (r5rs) is not working.
I have tested the VM only by the fib function above.
Please don't expect too much for now.

I'll hopefully get the VM usable during this summer.

Kei

[Evan Prodromou]

KN> I have checked in the current snapshot of my Guile VM to the
KN> CVS. If anyone is interested, please check it out and try it.

I'm very interested, but I hate seeing this work duplicated. Have you
taken a look at all at Gnu Lightning?

http://www.gnu.org/software/lightning/lightning.html

Shajnas al mi that it'd be faster and easier to work with this
existing JIT compiler than to make a whole nother one from scratch.

Then again, what do I know?

~ESP

[Marius Vollmer]

Post by Evan Prodromou
KN> I have checked in the current snapshot of my Guile VM to the
KN> CVS. If anyone is interested, please check it out and try it.
I'm very interested, but I hate seeing this work duplicated. Have you
taken a look at all at Gnu Lightning?

Please check out the guile-lightning module from CVS. It contains the
start for Lightning support in Guile. You can play arround with the
JIT assembler, but a lot details are missing to really integrate it
into the Guile run-time.

This is actually not at all in competition to a VM based design, since
Lightning is not really portable to every platform we want Guile to
run on. It is more like an additional performance boost for the most
popular platforms. Both will need a more or less sophisticated,
traditional Scheme compiler on top of them. This is an area where the
two approaches can share a lot of effort.

[ It can be argued that things like these is not really what Guile
needs most right now, that this is premature optimization, etc.
Well, yes. But I think better performance will allow us to write
more things in Scheme, which should speed up development
considerably.
]

[Evan Prodromou]

MV> Please check out the guile-lightning module from CVS. It
MV> contains the start for Lightning support in Guile. You can
MV> play arround with the JIT assembler, but a lot details are
MV> missing to really integrate it into the Guile run-time.

Yeah! That's for sure. guile-lightning seems totally bizarre to me,
actually. A user-level JIT? What's it for?

~ESP

[Marius Vollmer]

Post by Evan Prodromou
MV> Please check out the guile-lightning module from CVS. It
MV> contains the start for Lightning support in Guile. You can
MV> play arround with the JIT assembler, but a lot details are
MV> missing to really integrate it into the Guile run-time.
Yeah! That's for sure. guile-lightning seems totally bizarre to me,
actually. A user-level JIT? What's it for?

Well, you somehow need to write the JIT instructions, and I'm not
going to do that from C! ;)

I think this

((prolog 0)
(mov ret (scm 0))
(ret))

is much better than

jit_prolog (0);
jit_movr_l (JIT_RET, SCM_MAKINUM (0));
jit_ret ();

I make so much mistakes when coding JIT in C (because I don't have
memorized all the details) that the Scheme layer catches for me. For
example, I constantly write "jit_ldxr_l (JIT_R0, JIT_R0, offset)" when
I really mean "jit_ldxi_l". In Scheme its just "(ldx r0 r0 offset)"
and the details are filled in automatically.

[Neil Jerram]

Marius> [ It can be argued that things like these is not really
Marius> what Guile needs most right now, that this is premature
Marius> optimization, etc. Well, yes. But I think better
Marius> performance will allow us to write more things in Scheme,
Marius> which should speed up development considerably. ]

I agree. It seems to me that there is already a surprising number of
applications using Guile (GnuCash, Lilypond, TeXmacs, ...). For these
applications, improved performance means that they can write more of
the application in Scheme and so get more of the programming benefits
of the Scheme language.

Regards,
Neil

[Miroslav Silovic]

Post by Neil Jerram
I agree. It seems to me that there is already a surprising number of
applications using Guile (GnuCash, Lilypond, TeXmacs, ...). For these
applications, improved performance means that they can write more of
the application in Scheme and so get more of the programming benefits
of the Scheme language.

Don't forget SART. I'm still working on it. :)

And a raytracer that uses Scheme in the inner loop really needs good
performace (although not as much as you might think. Compiling the
Scheme with Hobbit improved the rendering time by only about 10%).

[Rob Browning]

Post by Neil Jerram
I agree. It seems to me that there is already a surprising number
of applications using Guile (GnuCash, Lilypond, TeXmacs, ...). For
these applications, improved performance means that they can write
more of the application in Scheme and so get more of the programming
benefits of the Scheme language.

ABSOLUTELY :>

[Rob Browning]

Post by Marius Vollmer
Please check out the guile-lightning module from CVS. It contains
the start for Lightning support in Guile. You can play arround with
the JIT assembler, but a lot details are missing to really integrate
it into the Guile run-time.

So I've been messing around with hobbit, and it does seem to work now,
though there are some things that would have to be fixed before anyone
would really want to use it.

However, in the end, all we (the gnucash people) really need is a
reasonable performance boost, and we do have a reasonable number of
(gnumatic) development hours we could spend on this. So I'm trying to
figure out if that time would be better spent helping with Lightning
and/or the VM since they sound like they may be superior long-term
solutions, and our efforts would benefit more people.

If, with reasonable effort, the VM or Lightning could be ready for
limited use in a month or so, then we'd probably like to help with
that. However, if it's going to be a lot longer than that, then I
suspect we should proably hold off, spend some time on other features
we need, and come back and help once we've finished those.

I'm also wondering if there's any technology we can lift from hobbit
to handle some of the compilation process needed when generating JIT
output.

So any sense of where things stand? If we can help, we'd like to.

Thanks

[Keisuke Nishida]

At 03 Apr 2001 13:46:24 -0500,

Post by Rob Browning
If, with reasonable effort, the VM or Lightning could be ready for
limited use in a month or so, then we'd probably like to help with
that. However, if it's going to be a lot longer than that, then I
suspect we should proably hold off, spend some time on other features
we need, and come back and help once we've finished those.

Hmm, if you don't need to support modules, like hobbit, you can use
my VM reasonably soon, by translating Scheme into GHIL directly:

(car x) => (@car x) ;; Don't look up `car' in modules

You can evaluate all expressions in `current-module' of Guile's
module system. Right now, my compiler basically produces the
following bytecode:

foo ->

load-module "user" ;; the module identified by the compiler
load-symbol "foo"
link ;; look up the symbol in the module above
variable-ref

You can modify it like this:

load-symbol "foo"
link/current-module ;; look up the symbol in the current module
variable-ref

All top-level variables are linked with the current module.
Probably this is all you have to do to support the current
module system.

Maybe I'll do it right now.

Kei

[Rob Browning]

Post by Keisuke Nishida
Hmm, if you don't need to support modules, like hobbit, you can use

Well, gnucash doesn't use modules at all (yet), though I was planning
to start switching us over to use them soon (now that I understand how
the current module system works better).

Post by Keisuke Nishida
load-symbol "foo"
link/current-module ;; look up the symbol in the current module
variable-ref

OK. I'll have to get more familiar with it before I'd know how to
deal with this.

Post by Keisuke Nishida
All top-level variables are linked with the current module.
Probably this is all you have to do to support the current module
system.
Maybe I'll do it right now.

Great. Let me know if it works. If I get time tomorrow, I'll see if
I can get your vm working here.

Thanks

[Keisuke Nishida]

At Tue, 03 Apr 2001 17:38:39 -0400,

Post by Keisuke Nishida
All top-level variables are linked with the current module.
Probably this is all you have to do to support the current
module system.
Maybe I'll do it right now.

Okay, it's now ready, somehow:

% guile-vm
Standard Scheme (R5RS + syntax-case) interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@user> (define foo 1)
***@user> foo
$1 = 1
***@user> (define (foo x y) (+ x y))
***@user> (foo 1 2)
$2 = 3
***@user> ,x foo
Disassembly of #<program 0x80ac6d8>:

args = 2 rest = 0 locals = 0

Bytecode:

0 local-ref 1
2 local-ref:0
3 object-ref 0
5 variable-ref
6 tail-call 2

Objects:

0 (+ . #<primitive-procedure +>)

Try with the following ~/.guile:

(define-module (guile-user))

(if (string=? (car (command-line)) "guile-vm")
(begin
(use-modules (system repl repl))
(use-modules (language r5rs expand))
(start-repl 'r5rs)
(quit)))

The r5rs translator uses the latest syntax-case expander,
which supports the `module' and `import' syntaxes (though
I haven't tried them yet.)

You can write your modules as follows:

---- fib.scm ---------------------------
(define-module (fib)
:use-module (system base language)
:use-module (language r5rs expand)
:export (fib))

(hacked-load-in "/somewhere/fib.gm" 'r5rs)
----------------------------------------

---- fib.gm ----------------------------
(define (fib n)
(if (< n 2) 1 (+ (fib (1- n)) (fib (- n 2)))))
----------------------------------------

Now, you can use it as a Guile module:

% guile
guile> (use-modules (fib))
guile> (fib 10)
$1 = 89
guile> fib
$1 = #<program 0x814db9c>

The procedure `hacked-load-in' above automatically compiles
the source file if it has been modified, save the compiled
code, and loads it. Thus, `fib' is a VM program, not a
Guile's closure, as shown above.

Hey, it works already, although it seems there are bugs,
and the load speed is very slow (this can be fixed).

Probably it's a good idea to use my VM this way for the moment.
Unfortunately, I'll be pretty busy during this month, so I'll
do this work in May. Do you want to wait, or hack my horrible
code?

My VM still lacks some important features, especially exception
handling. This must be implemented before it becomes ready for
use.

Kei

[Rob Browning]

That's great.

Though I'm having some trouble with current guile-core and guile-vm
from CVS. I followed your instructions, but I get the error included
below.

Post by Keisuke Nishida
Probably it's a good idea to use my VM this way for the moment.
Unfortunately, I'll be pretty busy during this month, so I'll do
this work in May. Do you want to wait, or hack my horrible code?

Not sure. Let me see once I get it working, then we can figure out
what would be most helpful.

Post by Keisuke Nishida
My VM still lacks some important features, especially exception
handling. This must be implemented before it becomes ready for
use.

So is it that the VM can't compile code that uses catch/throw, or that
byte-compiled code can't handle exceptions thrown to (or past) it?
The former would be OK, since we aren't using exceptions much in our
code right now. The latter though, would as you suggest, have to be
fixed first.

Thanks

$ LD_LIBRARY_PATH=/usr/local/opt/guile-cvs/lib PATH=/usr/local/opt/guile-cvs/bin:${PATH} guile-vm
Backtrace:
...
42 (let ((didit #f)) (dynamic-wind (lambda () #) (lambda () #) ...) ...)
43* [dynamic-wind #<procedure ()> #<procedure ()> #<procedure ()>]
44* [#<procedure ()>]
45* (let ((full #)) (if full (begin # #)))
46 (if full (begin (save-module-excursion (lambda () #)) (set! didit #t)))
47 (begin (save-module-excursion (lambda () #)) (set! didit #t))
48* [save-module-excursion #<procedure ()>]
49 (let (# #) (dynamic-wind # thunk #))
50 [dynamic-wind #<procedure ()> #<procedure ()> #<procedure ()>]
51* [#<procedure ()>]
52* [primitive-load "/usr/local/opt/guile-cvs/share/guile/site/system/base/mod$
53* (let ((module #)) (env-define (global-ref #) (quote module) ...) ...)
54* [env-define ...
55* [global-ref System::Base]
56 (letrec ((loop #)) (loop *root-package* (identifier->list identifier)))
...
57 [loop ...
58* [env-ref #<<multi-package> 808bf10> System]
59 [env-ref #<env 40385148> System]
...
60 [apply #<primitive-procedure env-ref> (#<env 40385148> System)]
61* [env-ref #<env 40385148> System]

ERROR: In procedure env-ref:
ERROR: Unbound variable in env: #<env 40385148>, System

[Keisuke Nishida]

At 03 Apr 2001 21:39:05 -0500,

Post by Rob Browning
Though I'm having some trouble with current guile-core and guile-vm
from CVS. I followed your instructions, but I get the error included
below.

Oops, sorry. Could you try the following patch?

--- guile-vm/module/system/base/module.scm.~1.1.~ Sat Mar 31 06:04:27 2001
+++ guile-vm/module/system/base/module.scm Tue Apr 3 23:38:48 2001
@@ -216,7 +216,3 @@
(let ((core (make-vmodule)))
(env-define *root-package* 'core core)
(hash-fold (lambda (s v d) (env-define core s v)) #f (builtin-bindings)))
-
-(let ((module (make-vmodule)))
- (env-define (global-ref 'System::Base) 'module module)
- (import-old-module! module (current-module)))

The real problem is that there is a hard-coded path in module.scm:

(define *root-package*
(make-multi-package '("/usr/local/share/guile/site")))

But since we are not going to use my module system, we can omit
this entirely. I'll apply a better solution some time later.

Post by Rob Browning

Post by Keisuke Nishida
My VM still lacks some important features, especially exception
handling. This must be implemented before it becomes ready for
use.

So is it that the VM can't compile code that uses catch/throw, or that
byte-compiled code can't handle exceptions thrown to (or past) it?
The former would be OK, since we aren't using exceptions much in our
code right now. The latter though, would as you suggest, have to be
fixed first.

Both. If you throw an exception to the VM, it goes through it.

[Rob Browning]

Post by Keisuke Nishida
(define *root-package*
(make-multi-package '("/usr/local/share/guile/site")))

I changed this for my location, and now it gets further, but dies
because make-env is not defined...

62* [initialize # #]
63 [%initialize-object # #]
64* (make-env)

/home/rlb/opt/guile-cvs/share/guile/site/system/base/module.scm:98:8: In expression (make-env):
/home/rlb/opt/guile-cvs/share/guile/site/system/base/module.scm:98:8: Unbound variable: make-env

[Keisuke Nishida]

At 04 Apr 2001 10:10:08 -0500,

Post by Rob Browning

Post by Keisuke Nishida
(define *root-package*
(make-multi-package '("/usr/local/share/guile/site")))

I changed this for my location, and now it gets further, but dies
because make-env is not defined...
62* [initialize # #]
63 [%initialize-object # #]
64* (make-env)

Hmm, make-env is defined in libguilevm.so, so it sounds like
dynamic-linking is not working correctly...

What happens if you type the following in guile?

% LD_LIBRARY_PATH=/usr/local/opt/guile-cvs/lib guile
guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> (define-module (system vm core))
#<directory (system vm core) 80837f0>
guile> make-vm
#<primitive-procedure make-vm>

And what if you try this?

% LD_LIBRARY_PATH=/usr/local/opt/guile-cvs/lib guile
guile> (dynamic-call "scm_init_envs" (dynamic-link "libguilevm.so"))
guile> (define-module (system base module))
#<directory (system base module) 80837b0>
guile> make-env
#<primitive-procedure make-env>

If this is not the case, maybe I'm doing something wrong.

[Rob Browning]

Post by Keisuke Nishida
Hmm, make-env is defined in libguilevm.so, so it sounds like
dynamic-linking is not working correctly...

(All of the following tests are using your latest CVS version
(as of the date of this message).)

Post by Keisuke Nishida
What happens if you type the following in guile?
% LD_LIBRARY_PATH=/usr/local/opt/guile-cvs/lib guile
guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> (define-module (system vm core))
#<directory (system vm core) 80837f0>
guile> make-vm
#<primitive-procedure make-vm>

$ LD_LIBRARY_PATH=/home/rlb/opt/guile-cvs/lib PATH=/home/rlb/opt/guile-cvs/bin:${PATH} guile
guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> (define-module (system vm core))
#<directory (system vm core) 8083e20>
guile> make-vm

Backtrace:
0* make-vm

<unnamed port>: In expression make-vm:
<unnamed port>: Unbound variable: make-vm
ABORT: (unbound-variable)
guile>

Post by Keisuke Nishida
And what if you try this?
% LD_LIBRARY_PATH=/usr/local/opt/guile-cvs/lib guile
guile> (dynamic-call "scm_init_envs" (dynamic-link "libguilevm.so"))
guile> (define-module (system base module))
#<directory (system base module) 80837b0>
guile> make-env
#<primitive-procedure make-env>

$ LD_LIBRARY_PATH=/home/rlb/opt/guile-cvs/lib PATH=/home/rlb/opt/guile-cvs/bin:${PATH} guile
guile> (dynamic-call "scm_init_envs" (dynamic-link "libguilevm.so"))
guile> (define-module (system base module))
#<directory (system base module) 8083e20>
guile> make-env

Backtrace:
0* make-env

<unnamed port>: In expression make-env:
<unnamed port>: Unbound variable: make-env
ABORT: (unbound-variable)
guile> (version)
"1.4.1"
guile>

[Keisuke Nishida]

At 04 Apr 2001 15:52:01 -0500,

Post by Rob Browning

Post by Keisuke Nishida
Hmm, make-env is defined in libguilevm.so, so it sounds like
dynamic-linking is not working correctly...

(All of the following tests are using your latest CVS version
(as of the date of this message).)

Sorry, I have changed the behavior of VM's init functions.
Could you try the following?

guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> make-vm
$1 = #<primitive-procedure make-vm>
guile> (dynamic-call "scm_init_envs" (dynamic-link "libguilevm.so"))
guile> make-env
$2 = #<primitive-procedure make-env>

I believe this should work. Otherwise, we have trouble with
dynamic linking.

[Rob Browning]

Post by Keisuke Nishida
Sorry, I have changed the behavior of VM's init functions.
Could you try the following?
guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> make-vm
$1 = #<primitive-procedure make-vm>

$ LD_LIBRARY_PATH=/home/rlb/opt/guile-cvs/lib PATH=/home/rlb/opt/guile-cvs/bin:${PATH} guile
guile> (dynamic-call "scm_init_vm" (dynamic-link "libguilevm.so"))
guile> make-vm

Backtrace:
0* make-vm

<unnamed port>: In expression make-vm:
<unnamed port>: Unbound variable: make-vm
ABORT: (unbound-variable)
guile>

I wonder if it's a namespace issue.

Maybe I've gotten something messed up here. I'm wondering if I
shouldn't try reinstalling guile-core and guile-vm from CVS, though
all that seemed to go right the first time...

[Keisuke Nishida]

I have updated the guile-vm module in CVS, and now it does not depend
on my new module system at all (I hope). Could you try it again?

Installation
------------

% ./autogen.sh
% configure
% make
% su
# make install
# ln -s module/{system,language} /usr/local/share/guile/site/

Add the following lines to your ~/.guile:

(if (string=? (car (command-line)) "guile-vm")
(begin
(use-modules (system repl repl))
(start-repl 'r5rs)
(quit)))

Example Session
---------------

% guile-vm
Standard Scheme (R5RS + syntax-case) interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@guile> (+ 1 2)
$1 = 3
***@guile> ,c -c (+ 1 2) ;; Compile into GLIL
(@asm (0 0 0 0)
(const 1)
(const 2)
(add)
(return))
***@guile> ,c -l (+ 1 2) ;; Compile into loadable code
0 make-int8:0 ;; 0
1 load-program #0
8 return

Bytecode #0:

0 make-int8:1 ;; 1
1 make-int8 2 ;; 2
3 add
4 return

***@guile> ,c (+ 1 2) ;; Compile and disassemble
Disassembly of #<program 0x810f75b>:

args = 0 rest = 0 locals = 0

Bytecode:

0 make-int8:1 ;; 1
1 make-int8 2 ;; 2
3 add
4 return

***@guile> (define (add x y) (+ x y))
***@guile> (add 1 2)
$2 = 3
***@guile> ,x add ;; Disassemble
Disassembly of #<program 0x8125f70>:

args = 2 rest = 0 locals = 0

Bytecode:

0 local-ref 1
2 local-ref:0
3 add
4 return

***@guile>

Write Modules
-------------

---- fib.scm ---------------------------
(define-module (fib)
:use-module (system vm load)
:export (fib))

(load/compile "fib.gsm")
----------------------------------------

---- fib.gsm ---------------------------
(define (fib n)
(if (< n 2)
1
(+ (fib (- n 1)) (fib (- n 2)))))
----------------------------------------

Now, expressions in fib.gsm are automatically compiled and
executed by the Guile VM:

% guile
guile> (use-modules (fib))
guile> (time (fib 30))
clock utime stime cutime cstime gctime
2.89 2.88 0.00 0.00 0.00 0.00
$1 = 1346269
guile> (define (fib n) (if (< n 2) 1 (+ (fib (- n 1)) (fib (- n 2)))))
guile> (time (fib 30))
clock utime stime cutime cstime gctime
26.05 25.01 0.17 0.00 0.00 14.33
$2 = 1346269

If you don't want to compile your code (e.g., for debugging purpose),
just change `load/compile' to `load'.

[Ariel Rios]

Post by Rob Browning
However, in the end, all we (the gnucash people) really need is a
reasonable performance boost, and we do have a reasonable number of
(gnumatic) development hours we could spend on this. So I'm trying to
figure out if that time would be better spent helping with Lightning
and/or the VM since they sound like they may be superior long-term
solutions, and our efforts would benefit more people.

Mi biggest fear with lightning is that it is platform specific and mine
(PPC)
is not suppored. This, however might be a good time to start hacking
the PPC port.

ariel

[Rob Browning]

Mi biggest fear with lightning is that it is platform specific.

Agreed, but no matter what approach we choose, there's going to be
plenty of work to do. IMO if it turns out that some of that work
needs to be on lightning, then so be it :>

[Evan Prodromou]

Mi biggest fear with lightning is that it is platform specific.

RB> Agreed, but no matter what approach we choose, there's going
RB> to be plenty of work to do. IMO if it turns out that some of
RB> that work needs to be on lightning, then so be it :>

It seems that the 2-step mechanism (compile to GLIL bytecode, then
interpret, JIT-compiling if supported on the platform) would
ameliorate the problem considerably.

~ESP

- --
Evan Prodromou
***@prodromou.san-francisco.ca.us

[Ariel Rios]

Post by Rob Browning
Agreed, but no matter what approach we choose, there's going to be
plenty of work to do. IMO if it turns out that some of that work
needs to be on lightning, then so be it :>

ok. Next week I am on holiday (yepee!) I can take some time
looking at lighting on PPC. I might be of help or I might not, who
knows? Where do I get latest code? I am looking at the release
available at alpha but I only get nice core dumps:

[***@soleil tests]$ ./printf
Illegal instruction (core dumped)
[***@soleil tests]$


ariel

[Rob Browning]

Post by Ariel Rios
ok. Next week I am on holiday (yepee!) I can take some time
looking at lighting on PPC. I might be of help or I might not, who
knows? Where do I get latest code? I am looking at the release
Illegal instruction (core dumped)

Hmm, I just built/ran the guile module, and that seemed to work. Come
to think of it, I don't recall installing lightning, so perhaps Marius
included the code in the guile module.

If you try the guile-lightning module from cvs, see if this works:

$ LD_LIBRARY_PATH=/home/rlb/opt/guile-cvs/lib\
PATH=/home/rlb/opt/guile-cvs/bin:${PATH} guile -l test.scm
guile> (numargs 'a 'b 'c)
3
guile>

[Marius Vollmer]

Post by Rob Browning
Hmm, I just built/ran the guile module, and that seemed to work. Come
to think of it, I don't recall installing lightning, so perhaps Marius
included the code in the guile module.

Yep, I did that. I had to modify Lightning already (bug fixes, new
features), so you need the version of Lightning included in
guile-lightning.

However, it will *only* work on i386 right now. I haven't added the
new instructions to the PowerPC and Sparc backends, and I haven't even
tried to make my changes portable yet. It's just fooling around. I
want to implement a very simple compiler for a very simple low-level
language, and maybe a peep-hole optimizer for JIT instructions.

If all works out nicely, one might then think about making it run on
PPC and Sparc. (And MIPS, because I'm getting myself an Agenda PDA
that runs Linux on a MIPS chip. 8-)

[Ariel Rios]

Post by Marius Vollmer
If all works out nicely, one might then think about making it run on
PPC and Sparc. (And MIPS, because I'm getting myself an Agenda PDA
that runs Linux on a MIPS chip. 8-)

So... Maybe I should wait to play with lighting...

ariel

[Marius Vollmer]

Post by Ariel Rios

Post by Marius Vollmer
If all works out nicely, one might then think about making it run on
PPC and Sparc. (And MIPS, because I'm getting myself an Agenda PDA
that runs Linux on a MIPS chip. 8-)

So... Maybe I should wait to play with lighting...

I don't know, if you feel like it, you can try to go along with the
things that I have done for the i386 and try them out on the PPC.

[Miroslav Silovic]

Post by Marius Vollmer
I don't know, if you feel like it, you can try to go along with the
things that I have done for the i386 and try them out on the PPC.

I'm interested, too (I might work on it if I find the time... but I
also have a bad case of curiosity to feed...). Do you have any docs
for the new instructions?

[Rob Browning]

Post by Marius Vollmer
I don't know, if you feel like it, you can try to go along with the
things that I have done for the i386 and try them out on the PPC.

For the past couple of days, I've had a half-formulated idea running
around in my head. I've been thinking about g-wrap, lightning,
libfcall, and libffi, and trying to figure out whether or not it might
be possible (and both reasonbly efficient and sufficiently portable)
to add some instructions to lightning that would allow on-the-fly
wrapping of C interfaces.

As it stands, g-wrap, given a spec file, dumps C wrappers
automatically to a .c file. For a large number of wrapped C
functions, this file can become quite large. so generating the wrapper
code as-needed could be advantageous, if it's not too expensive.

Further, with every call of a C-side function, the g-wrap wrappers
carefully check their arguments, but in cases where you have something
like this at the scheme level (presume all the functions called here
are g-wrapped C functions):

(let ((foo-obj (make-some-C-obj-ptr-of-type-foo x y z)))
(foo-frob! foo)
(foo-bar! foo)
(foo-baz! foo)
(foo-whatever foo)
foo)

there is *no* need to check the arguments to any of the function calls
since you know from the g-wrap prototype that the "make" function can
only return a foo ptr. So if the machine glue to call the C functions
was being generated at run-time (or at guile compile time), these
checks could be omitted for a substantial performance advantage,
especially in tight loops.

Another way to accomplish this would be to have g-wrap generate
checking and non-checking versions of all the functions and then just
call the right ones at the right times, but this would add more bloat,
and I thought the JIT/ffi/whatever solution was worth at least
considering for a moment.

I presume that for this to work, lightning would either have to have
instructions for performing the conversions, and the equivalent of a
c-side C "apply", or alternately, we'd need to integrate with
libffi/libfcall or similar.

The downside is that having g-wrap generate C wrapper code is
extremely portable, and one of these fancier solutions might be less
portable, unless we were willing to commit to keeping the
infrastructure working everywhere.

[Marius Vollmer]

Post by Rob Browning
For the past couple of days, I've had a half-formulated idea running
around in my head. I've been thinking about g-wrap, lightning,
libfcall, and libffi, and trying to figure out whether or not it might
be possible (and both reasonbly efficient and sufficiently portable)
to add some instructions to lightning that would allow on-the-fly
wrapping of C interfaces.

Hmm, might be worth a try. For what it's worth, you should be able to
use Lightning already to construct calls to C code:

:
(prepare 2)
(mov r0 "foo!")
(push r0)
(finish (subr "gtk_button_new_with_label"))
(retval r0)
(prepare 1)
(push r0)
(finish (subr "sgtk_wrap_gtkobj"))
(retval r0)
;; now a smob representing the new button is in R0
:

What is missing is a good way to do type checking and conversion.

Anyway, I'm not sure if run-time generation of wrappers is an
advantage. We are wrapping libraries that are batch-compiled, so why
not wrap them with another batch-compiled library?

Post by Rob Browning
Further, with every call of a C-side function, the g-wrap wrappers
carefully check their arguments, but in cases where you have something
like this at the scheme level (presume all the functions called here
(let ((foo-obj (make-some-C-obj-ptr-of-type-foo x y z)))
(foo-frob! foo)
(foo-bar! foo)
(foo-baz! foo)
(foo-whatever foo)
foo)
there is *no* need to check the arguments to any of the function calls
since you know from the g-wrap prototype that the "make" function can
only return a foo ptr.

I think you want to have a real compiler for this, that does real type
inference. In the midst of fixnum arithmetic and pair walking
optimizations it could also do some subr optimizations. In fact, the
compiler could maybe do the complete wrapping...

Post by Rob Browning
The downside is that having g-wrap generate C wrapper code is
extremely portable, and one of these fancier solutions might be less
portable, unless we were willing to commit to keeping the
infrastructure working everywhere.

Yes, I would really like to have a real compiler that compiles to
machine language, and have it work everywhere. But for the time
being, we should not make us dependent on clever hacks that do not
work everywhere.

I am quite confident that once we can demonstrate a compiler for, say,
the i386 platform, it will not be long before it gets ported to other
popular architectures and maybe at one time we can say that while we
need to keep Guile running on systems where we don't have a native
compiler, we don't need to care for performance on these systems.

However, this is all wild fantasy on my part. We should not make a
native code compiler another `Godot'. Do not wait for it.

[ Still, it _is_ great fun to work on a native compiler and, given GNU
Lightning, it is not really difficult, either. I now have a
prototype that efficiently compiles tail-calls and inner-function,
argument passing gotos. It does no register allocation yet, tho,
but that will come soon.

Here is an example that computes the sum of all integers upto N.
Note that there is no support for inlining of primitive operations
yet, so this isn't particularily fast.

(lambda-template (n)
(labels ((loop (i sum)
(if (invoke (global <=)
(local i) (local n))
(goto loop
(invoke (global +)
(local i) (quote 1))
(invoke (global +)
(local i) (local sum)))
(local sum))))
(goto loop (quote 1) (quote 0))))

=>

(beq l26 r1 4)
(prepare 1)
(mov r0 "some procedure")
(pusharg r0)
(finish (subr "scm_error_num_args_subr"))
l26
(mov r0 (scm 0))
(push r0)
(mov r0 (scm 1))
(push r0)
(b l27)
(b l28)
l27
(ldx r0 sp 12)
(push r0)
(ldx r0 sp 4)
(push r0)
(ld r0 (var #<variable 402fd570 name: <=
binding: #<primitive-procedure <=>>))
(mov r1 8)
(call (code #<codevector 80baa68>))
(mov r0 r0)
(beq l29 r0 (scm #f))
(ldx r0 sp 4)
(push r0)
(ldx r0 sp 4)
(push r0)
(ld r0 (var #<variable 40252878 name: +
binding: #<primitive-procedure +>>))
(mov r1 8)
(call (code #<codevector 80baa68>))
(mov r0 r0)
(stx 4 sp r0)
(mov r0 (scm 1))
(push r0)
(ldx r0 sp 4)
(push r0)
(ld r0 (var #<variable 40252878 name: +
binding: #<primitive-procedure +>>))
(mov r1 8)
(call (code #<codevector 80baa68>))
(mov r0 r0)
(stx 0 sp r0)
(b l27)
(b l30)
l29
(ldx r0 sp 4)
(ldx r2 sp 8)
(add sp sp 16)
(mov r1 4)
(jmp r2)
l30
(add sp sp 8)
(b l28)
l28

As you can see, a peephole pass is badly needed.
]

[Rob Browning]

Post by Marius Vollmer
Hmm, might be worth a try. For what it's worth, you should be able to

Interesting.

Post by Marius Vollmer
Anyway, I'm not sure if run-time generation of wrappers is an
advantage. We are wrapping libraries that are batch-compiled, so
why not wrap them with another batch-compiled library?

I'm not sure it is an advantage, but I thought it was worth
discussing. One potential win is space/code-density. In g-wrap, we
wrap about 313 functions, and the wrapper .o file alone turns out to
be 768K. Now perhaps I can shrink that by being smarter about what is
put in the wrappers, but off the top of my head, I don't think it'll
shrink much. With an on-line (or in-line) system, you only wrap the
things you need, where you need them (depending on how it's
implemented).

(One other thing I found interesting was that libfcall supports C side
closures. Not sure if it'd be useful for anything for us, though...)

Post by Marius Vollmer
I think you want to have a real compiler for this, that does real
type inference. In the midst of fixnum arithmetic and pair walking
optimizations it could also do some subr optimizations. In fact,
the compiler could maybe do the complete wrapping...

Right. That's what I was getting at. Mostly I was thinking of where
we want to go in the long run. Do we want to continue with an
off-line wrapper-processor and supplemental libwrapfoo.so files, or do
we want an on-line approach (is there any convincing benefit to
it?)...

Post by Marius Vollmer
I am quite confident that once we can demonstrate a compiler for, say,
the i386 platform, it will not be long before it gets ported to other
popular architectures

:>

Post by Marius Vollmer
[ Still, it _is_ great fun to work on a native compiler and, given GNU
Lightning, it is not really difficult, either. I now have a
prototype that efficiently compiles tail-calls and inner-function,
argument passing gotos. It does no register allocation yet, tho,
but that will come soon.

That's quite nice :>

[Rob Browning]

Post by Marius Vollmer
If all works out nicely, one might then think about making it run on
PPC and Sparc. (And MIPS, because I'm getting myself an Agenda PDA
that runs Linux on a MIPS chip. 8-)

Let me know how that goes. I've been wondering for a while if a
scheme PDA might be feasible/interesting :>

I played with lispme for a while, but the first-generation palm
hardware I had was really flaky and fairly limited.

[Ariel Rios]

Post by Rob Browning
Let me know how that goes. I've been wondering for a while if a
scheme PDA might be feasible/interesting :>

Ah! Life has coincidences! I am working in a project at
my University called M-Tec (Mobile Tec) that tends to
bring palm pilot technology to th e campus comunity.
It would be great to be able to use guile...

Post by Rob Browning
I played with lispme for a while, but the first-generation palm
hardware I had was really flaky and fairly limited.

At this moment, lispme is my only choice (lisp choice that is
I do not want to do C, Java or python) is it good enough
to do simple apps?

ariel

[Rob Browning]

Post by Ariel Rios

Post by Rob Browning
I played with lispme for a while, but the first-generation palm
hardware I had was really flaky and fairly limited.

At this moment, lispme is my only choice (lisp choice that is
I do not want to do C, Java or python) is it good enough
to do simple apps?

I believe so, but I didn't do a whole lot with it. I know that in
addition to computation, you can manipulate graphics, create apps with
icons, etc.

[Marius Vollmer]

Post by Rob Browning
So any sense of where things stand? If we can help, we'd like to.

I can't really speak for the VM, but what I would like to see is some
investigation of existing `butch' Scheme or Lisp compilers. In what
way could they be used with Guile? For example, can Stalin be
modified to compile modules in a non-closed world, emitting the simple
mid-level language for the compiler in guile-lightning?

Guile-lightning has or will have only `low-level' expertise. That is,
it knows for example how to compile parameter passing gotos into stack
and register shuffling code, and how to inline primitive operations
once it has been told that it should do so. It does not know how to
represent general closures or when to is safe to inline a call to a
function. It does not know about type propagation, but you can tell
it to use special versions of primitive operations that omit certain
type checks.

Thus, guile-lightning should take care of the machine specific
details, presenting a language to the upper layers that should be as
simple to target as C. Simpler, in fact, since it takes care of the
idiosyncratic control flow features of Scheme.

I'm still thinking about where the right point is to interface with a
VM. Do we (ultimately) want a system that has either a native
compiler or a VM? Or a system that always has a VM, and on some
platforms also a native compiler? I'm not sure yet. In any case,
having the compiler in guile-lightning generate byte-codes instead of
JIT instructions seems quite reasonable. I kinda like the dual
approach: a VM is useful even when we have a native compiler (for
debugging and compactness), and when we fuse the two on such a low
level, there should be little problem with maintaining them.


Anyway, I'm going to add register allocation to guile-lightning
now... this is soo much fun. :-)

[Evan Prodromou]

MV> I'm still thinking about where the right point is to interface
MV> with a VM. Do we (ultimately) want a system that has either a
MV> native compiler or a VM? Or a system that always has a VM,
MV> and on some platforms also a native compiler?

Ding ding ding.

This seems to be the Right Thing: compile to VM code, and then
JIT-compile to Lightning if possible. In pseudo,

byte_code = byte_code_compile(raw_code);

#ifdef __HAVE_LIGHTNING__

mixed_native_and_byte_code = jit_compile(byte_code);
result = jit_run(mixed_native_and_byte_code);

#else

result = byte_code_run(byte_code);

#endif

~ESP

- --
Evan Prodromou
***@prodromou.san-francisco.ca.us

[Marius Vollmer]

Post by Evan Prodromou
This seems to be the Right Thing: compile to VM code, and then
JIT-compile to Lightning if possible.

I was more thinking about: compile to native using Lightning (ignoring
its JIT focus), or to bytecodes if needed. That is, we would have a
(run-time) retargetable compiler that can either generate native code
via Lightning, or bytecodes for a VM. I don't think bytecodes that
are tuned for performance are a good intermediate language.

[Rob Browning]

Post by Marius Vollmer
I can't really speak for the VM, but what I would like to see is
some investigation of existing `butch' Scheme or Lisp compilers. In
what way could they be used with Guile? For example, can Stalin be
modified to compile modules in a non-closed world, emitting the
simple mid-level language for the compiler in guile-lightning?

This is a good point. I've been playing with hobbit, but I can't say
I understand it to any notable extent. I haven't really looked at
what it's doing internally.

I could also ask Siskind for his opinion. I've spoken with him off an
on over the past couple of years about various things, including the
possibility of using Stalin in parallel with other languages in a very
"hacked" setup just to gain performance. I'll try and send him a
message shortly and see if he has any thoughts.

Post by Marius Vollmer
I kinda like the dual approach: a VM is useful even when we have a
native compiler (for debugging and compactness), and when we fuse
the two on such a low level, there should be little problem with
maintaining them.

This sounds pretty good, though I wonder if applying lightning after
moving to the byte code stage might mean losing information that would
allow jit optimizations that wouldn't make sense for the VM.
i.e. does the original source contain information that would suggest
one kind of optimization for byte-codes and another for JIT. I'm not
sure this is a serious concern, though...

[Marius Vollmer]

Post by Rob Browning
I could also ask Siskind for his opinion. I've spoken with him off an
on over the past couple of years about various things, including the
possibility of using Stalin in parallel with other languages in a very
"hacked" setup just to gain performance. I'll try and send him a
message shortly and see if he has any thoughts.

Nice. (You might even ask him if he would assign the copyright to
Stalin to the FSF, ahem.)

Post by Rob Browning

Post by Marius Vollmer
I kinda like the dual approach: a VM is useful even when we have a
native compiler (for debugging and compactness), and when we fuse
the two on such a low level, there should be little problem with
maintaining them.

This sounds pretty good, though I wonder if applying lightning after
moving to the byte code stage might mean losing information that would
allow jit optimizations that wouldn't make sense for the VM.

I'm not so much thinking about "Scheme -> VM -> native", more about

-> native
Scheme --> some low level stuff -|
-> VM

[Rob Browning]

Post by Marius Vollmer
Nice. (You might even ask him if he would assign the copyright to
Stalin to the FSF, ahem.)

OK.

I'm actually the one who badgered him in the first place about
changing the license to GPL. I don't know what his opinion about
copyright assignment would be, but now that you mention it, that might
be a problem since I believe NEC would have to be involved as well.
As his employers, I seem to recall that they were involved in the
agreement to change the license. (My thanks to NEC for agreeing BTW).

Post by Marius Vollmer
I'm not so much thinking about "Scheme -> VM -> native", more about
-> native
Scheme --> some low level stuff -|
-> VM

Sounds good, and we can have whatever annotations are needed in the
"low-level stuff" in order to do appropriate optimizations. One thing
I'd wondered about was whether or not we might want the "low level
stuff" here to actually be a byte code (though perhaps not the VM one)
or some other arch-independant disk writable format.

If we could have a post-compilation (i.e. after the really expensive
analysis) representation that was arch-independent, but could be
cheaply converted to VM input or assembly (via lightning) at load
time, we could have both speed and portability. This format might
have to have more information that just VM byte codes, though, for the
reasons mentioned above...

Dunno, but maybe worth thinking about.

[Keisuke Nishida]

At 09 Apr 2001 21:52:33 -0500,

Post by Rob Browning

Post by Marius Vollmer
I'm not so much thinking about "Scheme -> VM -> native", more about
-> native
Scheme --> some low level stuff -|
-> VM

Sounds good, and we can have whatever annotations are needed in the
"low-level stuff" in order to do appropriate optimizations. One thing
I'd wondered about was whether or not we might want the "low level
stuff" here to actually be a byte code (though perhaps not the VM one)
or some other arch-independant disk writable format.
If we could have a post-compilation (i.e. after the really expensive
analysis) representation that was arch-independent, but could be
cheaply converted to VM input or assembly (via lightning) at load
time, we could have both speed and portability. This format might
have to have more information that just VM byte codes, though, for the
reasons mentioned above...
Dunno, but maybe worth thinking about.

This is exactly what I'm thinking of now. I call the "low level stuff"
GLIL, which currently looks like this:

(@asm (0 0 0 0)
(const 1)
(const 2)
(add)
(return))

Certainly, we can change the format and add more information.
What kind of information should be included? I just can't
decide what is the best intermediate representation right now.

Keisuke

[Evan Prodromou]

Post by Rob Browning
Sounds good, and we can have whatever annotations are needed in
the "low-level stuff" in order to do appropriate optimizations.

KN> This is exactly what I'm thinking of now. I call the "low
KN> level stuff" GLIL, which currently looks like this:

KN> (@asm (0 0 0 0) (const 1) (const 2) (add) (return))

KN> Certainly, we can change the format and add more information.
KN> What kind of information should be included? I just can't
KN> decide what is the best intermediate representation right now.

See, I'd probably say that the value of having a common intermediate
representation (and really, isn't it pretty much a virtual machine
language? Or am I nuts?) far outweighs the optimization advantages of
having separate representations.

SURE, you could possible squeak out 1-2% better performance by
maintaining lots of high-level information (or even programmer hints)
along the way, but then again having *any* compilation functionality
sooner rather than later seems much more important than
super-optimizing at the design phase.

~ESP

[Rob Browning]

Post by Evan Prodromou
See, I'd probably say that the value of having a common intermediate
representation (and really, isn't it pretty much a virtual machine
language? Or am I nuts?) far outweighs the optimization advantages
of having separate representations.

Well, actually, all I had been thinking was that if a common language
is feasible (and for simplicity's sake I hope it is), that language
might need to represent a superset of what each of the backends need
-- containing markup that might mean nothing to the vm but be
important to lightning, and vice-versa. Presumably such markup would
be fairly rare, but I suspect there may be situations where it could
have a big impact.

Post by Evan Prodromou
SURE, you could possible squeak out 1-2% better performance by
maintaining lots of high-level information (or even programmer hints)
along the way, but then again having *any* compilation functionality
sooner rather than later seems much more important than
super-optimizing at the design phase.

Agreed.

[Marius Vollmer]

Certainly, we can change the format and add more information. What
kind of information should be included? I just can't decide what is
the best intermediate representation right now.

What about something along these lines (this is from guile-lightning,
and in flux):

;; The language:
;;
;; Fully parenthesized prefix syntax with nothing but special forms.
;;
;; Special forms are
;;
;; - (lambda-template (ARGS... :rest REST :env ENV) BODY...)
;;
;; ARGS, REST and ENV are symbols and BODY is a list of expressions.
;; Both `:rest' and `:env' are optional.
;;
;; Makes a closure template, argument ENV refers to the
;; environment given to MAKE_CLOSURE, below. Closure templates
;; can not be invoked. Rest argument denoted by `:rest'.
;;
;; [ This is not the most expressive way to handle arguments. Maybe
;; have explicit (ref-arg N) and (args-list N) etc forms? ]
;;
;; - (make-closure TEMPLATE ENV)
;;
;; Make a closure from a closure template and an environment. A
;; closure can be invoked. ENV can be anything.
;;
;; - (if TEST THEN ELSE)
;;
;; The usual conditional. Else clause is mandatory.
;;
;; - (quote OBJ)
;;
;; Refers to the Scheme object OBJ, which can be anything.
;;
;; - (local SYMBOL)
;;
;; Retrieve the value of the local variable SYMBOL, as established
;; by LABELS, FUNCTIONS or LAMBDA-TEMPLATE.
;;
;; - (set-local SYMBOL VAL)
;;
;; Set the value of the local variable SYMBOL, as established
;; by LABELS, FUNCTIONS or LAMBDA-TEMPLATE.
;;
;; Note that SET-LOCAL does not interact correctly with call/cc.
;; The values of locals are copied by call/cc and any changes to
;; them between the call to call/cc and when the continuation is
;; invoked are lost.
;;
;; - (global SYMBOL)
;;
;; Retrieve the value of the global variable that is named by SYMBOL
;; in the current module. (Current at the time of linking.)
;;
;; - (invoke PROC ARGS...)
;;
;; Invokes PROC with ARGS. PROC can evaluate to any Scheme object.
;; However, when PROC is of the form `(global VAR)', we cooperate
;; with the module system to generate a more efficient calling
;; sequence.
;;
;; - (inline OP ARGS...)
;;
;; Generate inline code for OP and ARGS. OP must have been
;; registered with the compiler previously as an inline-operator.
;; Typical OPs are `+', `car', etc.
;;
;; - (labels ((LABEL (ARGS...) BODY) ...) BODY)
;;
;; Establish labels that can be jumped to with GOTO, which see. The
;; labels are visible within all of the BODYs in the labels form.
;; The continuation of each BODY is the continuation of the LABELS
;; form.
;;
;; ARGS can be of the form (SYMBOL :reg N) to indicate a register
;; preference level. Just SYMBOL is equivalent to (SYMBOL :reg 0).
;;
;; - (goto LABEL ARGS...)
;;
;; Transfer control to LABEL, passing it ARGS.
;;
;; - (functions ((FUNC (ARGS...) BODY) ...) BODY)
;;
;; Like `labels', but you need to use `call' instead of `goto'.
;;
;; - (call FUNC args...)
;;
;; Invoke FUNC, passing it ARGS.
;;
;;
;; Example:
;;
;; (lambda-template (n)
;; (if (inline < (local n) (quote 2))
;; (quote 1)
;; (inline + (invoke (global fib) (inline - (local n) (quote 2)))
;; (invoke (global fib) (inline - (local n) (quote 1))))))
;;
;;
;; Note that there is no `let' equivalent. Use `labels' instead.
;;
;; A `functions' form where all `calls' appear in tail positions is
;; semantically equivalent to a `labels' form, but the compiler
;; doesn't detect this. It still emits code to handle the general
;; case. You have to help it by explicitely using a `labels' form
;; when possible. This might change in the future, but right now,
;; `functions' isn't even implemented at all.
;;
;; In general, this compiler does no optimizations that could have
;; been performed on its input source. The expertise of this compiler
;; should be argument shuffling to implement tail calls and
;; parameter-passing gotos, branch optimizations, (simple) register
;; allocation, and inlining of certain operations like fixnum
;; arithmetic and cons cell walking.
;;
;; Things like closure-conversion, removal of explicit lambdas etc are
;; left to the upper layers.

I think I'd prefer to work with such a language (compared to a stack
based language like GLIL). The stack is already too explicit, and
probably interferes with register allocation. On the other hand
generating stack-code from this should be very easy.

I think this language is easy to target for Scheme. `functions' is
intended for known functions that need no heap closure, and `labels'
is for known functions that only tail call each other, i.e., which
don't need a closure at all.

I'm not convinced that this is all that is needed, but it feels like a
good start. I want to play a bit (or two) with the Larceny compiler
as a front end for the guile-lightning compiler.

[Michael Livshin]

Post by Marius Vollmer

Certainly, we can change the format and add more information. What
kind of information should be included? I just can't decide what is
the best intermediate representation right now.

What about something along these lines (this is from guile-lightning,

looks very very nice!

some mostly idle/academic questions:

1) I don't see any `capture-continuation'/`reinstate-continuation'
primitives. I assume this is because continuation primitives are
packaged up as "known functions", right?

if so, this is quite appropriate for the current Guile runtime
model, but other possible models may require that these things be
more explicit. (it's entirely possible that I don't know what I'm
talking about here, but hey...).

2) you do explicitly say that this IL is Scheme-oriented. what would
be the minimal set of additions to make it suitable as a target for
an Elisp compiler? for CL? for Python?

Post by Marius Vollmer
I'm not convinced that this is all that is needed, but it feels like a
good start. I want to play a bit (or two) with the Larceny compiler
as a front end for the guile-lightning compiler.

cool!

[ btw, it looks like the next version of MzScheme is going to look
not entirely unlike Guile + Godot. would be interesting to play
with that, too... ]

[Rob Browning]

Post by Michael Livshin
2) you do explicitly say that this IL is Scheme-oriented. what would
be the minimal set of additions to make it suitable as a target for
an Elisp compiler? for CL? for Python?

IMO, this would be *very* nice, and perhaps having a language neutral
IL is more likely to succeed than other-language->guile front ends.

I think it would be wonderful if we ended up with a really good hybrid
IL -> byte-code/jit-machine code compiler that could host a variety of
languages. Sounds like a tall order, though :>

[Keisuke Nishida]

At 12 Apr 2001 11:45:40 -0500,

Post by Rob Browning

Post by Michael Livshin
2) you do explicitly say that this IL is Scheme-oriented. what would
be the minimal set of additions to make it suitable as a target for
an Elisp compiler? for CL? for Python?

IMO, this would be *very* nice, and perhaps having a language neutral
IL is more likely to succeed than other-language->guile front ends.
I think it would be wonderful if we ended up with a really good hybrid
IL -> byte-code/jit-machine code compiler that could host a variety of
languages. Sounds like a tall order, though :>

And this is my plan, as I said before. I want to execute at least
Scheme and ML on my VM. I'm getting feelings that I really need a
statically typed language when I implement something like a compiler.

Although translating a target language -> IL -> compiled code looks
like a long path, we don't need a literal translation in practice.
What we really need is a common internal parse tree. Although
I often write the list representation of GLIL (i.e., "(@asm () ...)"),
no such list is actually produced during compilation. Therefore,
having an IL does not make additional cost (I guess).

[Marius Vollmer]

Post by Michael Livshin
1) I don't see any `capture-continuation'/`reinstate-continuation'
primitives. I assume this is because continuation primitives are
packaged up as "known functions", right?

Hmm, not even known functions. I was thinking about using the
existing Guile mechanism for continuations, which uses a jmpbuf and a
stack copy. That is, "call/cc" would be a normal function that is
invoked via "invoke". The compiler would only have to deal with
continuations in that it will have to perform `assignment
elimination', that is, it would have to put local variables that are
mutated into the heap (as part of a closure that it has to create
anyway, or explicitely).

In any case, it is probably possible to implement call/cc via some
"inline" op. These inline ops are supposed to be real low level and
expressed in the target language of the compiler.

Post by Michael Livshin
2) you do explicitly say that this IL is Scheme-oriented. what would
be the minimal set of additions to make it suitable as a target for
an Elisp compiler? for CL? for Python?

Interesting question. I don't know. The `labels' thing should be
general enough to be usable for all local control structures known to
man. It should also be efficient.

For Elisp, the lexical scoping is not of much help, I think, but
dynamic scoping, buffer local variables and stuff should be easy to
add. Either as "inline" ops, or as new special forms that are more or
less trivial to implement.

CL? I can't think of any specific problem right now.

I don't know enough about Python. Maybe it needs special support for
its OO features, but probably not.

[Keisuke Nishida]

At 11 Apr 2001 23:49:57 +0200,

Post by Marius Vollmer

Certainly, we can change the format and add more information. What
kind of information should be included? I just can't decide what is
the best intermediate representation right now.

What about something along these lines (this is from guile-lightning,

Looks interesting, and this is more like the internal form of my GHIL,
which is a structured language like yours. After parsing and macro
expansion, GHIL is reduced to a simpler form like this language.
(After then, it is converted into GLIL.)

So, maybe I should stop transformation at the end of the GHIL stage.

Post by Marius Vollmer
;; - (lambda-template (ARGS... :rest REST :env ENV) BODY...)
;;
;; ARGS, REST and ENV are symbols and BODY is a list of expressions.
;; Both `:rest' and `:env' are optional.

How do you use ENV? Are you adding `env-ref' or something?
Is it dynamic or static? (i.e., when do you look up?)

Post by Marius Vollmer
;; Makes a closure template, argument ENV refers to the
;; environment given to MAKE_CLOSURE, below. Closure templates
;; can not be invoked. Rest argument denoted by `:rest'.

(By the way, a closure template can be invoked in my VM
when it is known that the closure template does not contain
any environment references. Actually, I do not distinguish
a closure template from a closure; one is a closure without
an environment, and the other is a closure with an environment.
Closures without environments are constants:

***@guile> (define (foo) (lambda () 1))
***@guile> ,disassemble foo
Disassembly of #<program foo>:

0 object-ref 0 ;; #<program 80f7e26>
2 return
)

Post by Marius Vollmer
;; - (global SYMBOL)
;;
;; Retrieve the value of the global variable that is named by SYMBOL
;; in the current module. (Current at the time of linking.)

Are you going to look up global variables at loading time?
If so, you can't do global analysis at compile time, can you?
I'd prefer an explicit reference as follows:

(module-ref MODULE-NAME SYMBOL)
(module-set MODULE-NAME SYMBOL VALUE)

(But in my current VM, MODULE-NAME is just ignored.)

Post by Marius Vollmer
;; A `functions' form where all `calls' appear in tail positions is
;; semantically equivalent to a `labels' form, but the compiler
;; doesn't detect this. It still emits code to handle the general
;; case. You have to help it by explicitely using a `labels' form
;; when possible. This might change in the future, but right now,
;; `functions' isn't even implemented at all.

Do you really need the `functions' form? You distinguish global
function calls (`invoke') from local function calls (`call'),
which complicates the system, I think. But maybe it's worth
doing. Hmm, I need to think about it.

Post by Marius Vollmer
I think I'd prefer to work with such a language (compared to a stack
based language like GLIL). The stack is already too explicit, and
probably interferes with register allocation. On the other hand
generating stack-code from this should be very easy.

I agree with that. We should be able to combine our efforts
after some experiences.

Keisuke

[Marius Vollmer]

Post by Keisuke Nishida

Post by Marius Vollmer
What about something along these lines (this is from guile-lightning,

Looks interesting, and this is more like the internal form of my GHIL,
which is a structured language like yours. After parsing and macro
expansion, GHIL is reduced to a simpler form like this language.

I see. I haven't really looked at your VM yet (shame, shame). It's
too damn much fun playing with assembly language in an interactive
system.

Post by Keisuke Nishida
So, maybe I should stop transformation at the end of the GHIL stage.

Yes. The language that I presented (looks like I need a name for it,
hmm, what about Glow, in lieu of Lightning ;-) will need quite a
sophisticated front-end, I think, since Glow is very limited.
Especially closure conversion (with lambda lifting etc) is likely a
non-trivial thing, and then of course type inference to be able to
inline simple operations.

Post by Keisuke Nishida

Post by Marius Vollmer
;; - (lambda-template (ARGS... :rest REST :env ENV) BODY...)
;;
;; ARGS, REST and ENV are symbols and BODY is a list of expressions.
;; Both `:rest' and `:env' are optional.

How do you use ENV? Are you adding `env-ref' or something?
Is it dynamic or static? (i.e., when do you look up?)

I'm not sure about :env yet. ENV will probably be just another
argument for the function that can be accessed via `(local ENV)'.

Lookup using ENV will be explicit, like

(inline env-ref (local env) 2 3)

which could mean something like `get the value in slot 3 of frame 2 in
the environment'. How the environment is represented is of no concern
to this compiler. Maybe it should, but I try to ignore this issue for
now.

Post by Keisuke Nishida

Post by Marius Vollmer
;; - (global SYMBOL)
;;
;; Retrieve the value of the global variable that is named by SYMBOL
;; in the current module. (Current at the time of linking.)

Are you going to look up global variables at loading time?

Yes.

Post by Keisuke Nishida
If so, you can't do global analysis at compile time, can you?

Not without help from the code that is analysed, no.

I can imagine using declarations to guide the compiler. For example,
we could have a way to declare certain bindings as being constant:

(define foo ...)
(declare foo (constant))

This would allow the compiler to notice that the value of this binding
is not going to change (mostly, see below), and to arrange things so
that the linker can hard-code an efficient way to, say, invoke `foo'
at load time. This might be quite noticeable with `subrs'. Or not.

Although such a binding is declared constant, this wouldn't mean that
it doesn't change. You will not be able to change it programmatically
via `set!', but the user can do so interactively. The module system
will provide notifications about this to the linker, so that the
hard-coding mentioned above can be redone.

For example, the compiler could notice that `union' constantly refers
to a compiled Scheme procedure and could generate code to invoke it
directly. The linker will then patch in the real locations of the
actual code and the environment whenever the `union' binding changes.
When it changes to something different from a compiled Scheme
procedure, the linker could patch in trampoline code (and emit a
warning).

The user might also put a declaration that allows the compiler to
inline certain functions. This means that you can't change the
definition of these functions without recompiling the code that uses
them, of course. This declaration need not be colocated with the
definition of the functions being inlined. We might have somthing
like this, for example

(define-module (bla)
:use-module (ice-9 core))

(declare (inline-module (ice-9 core)))

This would allow the compiler to inline all functions defined in the
(ice-9 core) module. It wont necessarily inline the whole function
but it will very likely generate a specialiced calling sequence that
will be incompatible when the arguments of the `inlined' function
changes. (If the arguments change incompatably, we can just patch in
a stub that throws an error, tho.)

This would also be the mechanism that allows the compiler to figure
out that `+' really refers to the standard addition procedure, all the
time.

Post by Keisuke Nishida
(But in my current VM, MODULE-NAME is just ignored.)

Post by Marius Vollmer
;; A `functions' form where all `calls' appear in tail positions is
;; semantically equivalent to a `labels' form, but the compiler
;; doesn't detect this. It still emits code to handle the general
;; case. You have to help it by explicitely using a `labels' form
;; when possible. This might change in the future, but right now,
;; `functions' isn't even implemented at all.

Do you really need the `functions' form? You distinguish global
function calls (`invoke') from local function calls (`call'),
which complicates the system, I think.

I think unknown functions are sufficiently different from known local
functions. For known local functions, we can have a customized
calling convention, while for unknown functions, we must adhere to a
standard convention. Incidentally, all stack manipulation, register
allocation, and register spilling is concentrated in the calling
sequences in the guile-lightning compiler. That is, register
allocation is done by assigning certain arguments of known local
functions to registers.

There might actually be a need for a third kind of known local
functions: ones that are known local, but have no references to free
variables (after closure conversion). They would be used for known
local functions that are closed over by escaping lambdas. We want to
have a custom calling convention for them, but we don't know at
compile time in what environment they will execute.

[Rob Browning]

Post by Marius Vollmer
I'm not convinced that this is all that is needed, but it feels like a
good start. I want to play a bit (or two) with the Larceny compiler
as a front end for the guile-lightning compiler.

FWIW, I sent a mail to Siskind to see if he thinks stalin might be at
all suitable. I kinda suspect that stalin's assumptions and target
envt are too different, but we'll see what he says.

Does anyone here know enough about hobbit to know if it has useful
algorithms? I haven't poked around in the docs yet much.

[Marius Vollmer]

Post by Rob Browning
FWIW, I sent a mail to Siskind to see if he thinks stalin might be at
all suitable. I kinda suspect that stalin's assumptions and target
envt are too different, but we'll see what he says.

Yes, thanks. What I would want from Stalin is the aggressive type
inference, and maybe its expertise in representing closures.

[Keisuke Nishida]

At 31 Mar 2001 22:29:56 -0800,

Post by Evan Prodromou
KN> I have checked in the current snapshot of my Guile VM to the
KN> CVS. If anyone is interested, please check it out and try it.
I'm very interested, but I hate seeing this work duplicated. Have you
taken a look at all at Gnu Lightning?
http://www.gnu.org/software/lightning/lightning.html
Shajnas al mi that it'd be faster and easier to work with this
existing JIT compiler than to make a whole nother one from scratch.

My VM lacks a JIT compiler; it's just a simple bytecode interpreter.
If you want a JIT or a real compiler, you could convert the intermediate
form (GLIL) of my compiler into machine code, using Marius's Lightning
module. My work will put more emphasis on the higher layer of the
compiler, including a module system, a type system, and optimization.

Kei

[Rob Browning]

Post by Keisuke Nishida
My VM lacks a JIT compiler; it's just a simple bytecode interpreter.
If you want a JIT or a real compiler, you could convert the
intermediate form (GLIL) of my compiler into machine code, using
Marius's Lightning module. My work will put more emphasis on the
higher layer of the compiler, including a module system, a type
system, and optimization.

And I think there's also the possibility that a hybrid, combining JIT
and byte-code might be a really interesting approach. For some parts
of your code, the memory compression of bytecoding might be a big win,
and for the tightest inner loops, you might want Lightning.

I wonder if you could hook up the system with a profiler so that
during development guile could identify the critical sections and try
each approach, eventually choosing the one that's fastest. Then you
might need a way to annotate your code so that when you load normally,
guile will choose the right one by default.

Another interesting possibility would be a run-time profiler that
compiled a cached subset of the most frequently called functions --
this might be good for a guile webserver or database that computed in
"phases".

[Keisuke Nishida]

At 03 Apr 2001 13:10:28 -0500,

Post by Rob Browning
And I think there's also the possibility that a hybrid, combining JIT
and byte-code might be a really interesting approach. For some parts
of your code, the memory compression of bytecoding might be a big win,
and for the tightest inner loops, you might want Lightning.

That's interesting. Probably we can do it in the long run.

Post by Rob Browning
I wonder if you could hook up the system with a profiler so that
during development guile could identify the critical sections and try
each approach, eventually choosing the one that's fastest. Then you
might need a way to annotate your code so that when you load normally,
guile will choose the right one by default.
Another interesting possibility would be a run-time profiler that
compiled a cached subset of the most frequently called functions --
this might be good for a guile webserver or database that computed in
"phases".

Possible, but I think we shouldn't think about this sort of
optimization right now.

Kei

[Evan Prodromou]

KN> My VM lacks a JIT compiler; it's just a simple bytecode
KN> interpreter. If you want a JIT or a real compiler, you could
KN> convert the intermediate form (GLIL) of my compiler into
KN> machine code, using Marius's Lightning module. My work will
KN> put more emphasis on the higher layer of the compiler,
KN> including a module system, a type system, and optimization.

Yeah, I think that makes a lot of sense, actually. So the model would
be, compile to GLIL bytecode, and then interpret. If you can do some
Lightning JIT compiling during interpretation, cool. If not, heck,
you're still ahead of the game.

Sounds like a good and fairly standard strategy.

~ESP

- --
Evan Prodromou
***@prodromou.san-francisco.ca.us

[Rob Browning]

We've grabbed and applied your profiling patch here to guile 1.4, and
we've been using it to try and profile some things that need to be
sped up. However, I'm a little suspicious of its results. As an
example, there are cases where a given function is listed as being
called 32000 times, yet one of it's sub-functions (that is called
unconditionally) shows up as *never* being called.

Any idea what might be wrong?

Thanks

[Keisuke Nishida]

At 04 Apr 2001 16:36:15 -0500,

Post by Rob Browning
We've grabbed and applied your profiling patch here to guile 1.4, and
we've been using it to try and profile some things that need to be
sped up. However, I'm a little suspicious of its results. As an
example, there are cases where a given function is listed as being
called 32000 times, yet one of it's sub-functions (that is called
unconditionally) shows up as *never* being called.
Any idea what might be wrong?

Is the sub-function not shown at a tail position? If so, probably
I was missing recording tail calls.

[Rob Browning]

Post by Keisuke Nishida
Is the sub-function not shown at a tail position? If so, probably
I was missing recording tail calls.

Here's the function. According to the profiler, it's called 32094
times, but also according to the profiler, gnc:html-style-table-fetch
is never called, but if I put a print statement just before the set!,
I see that it's called all the time.

Any idea what might be wrong and how I might be able to fix it?
Thanks.

;; Code is a little ugly b/c optimizing tests for execution speed.
(define (gnc:html-document-fetch-markup-style doc markup)
(let ((style-info #f)
(style-stack (gnc:html-document-style-stack doc)))

(if (null? style-stack)
(or style-info (gnc:make-html-markup-style-info))
(begin
(set! style-info (gnc:html-style-table-fetch (car style-stack)
(cdr style-stack)
markup))
(or style-info (gnc:make-html-markup-style-info))))))

[Keisuke Nishida]

At 04 Apr 2001 17:47:01 -0500,

Post by Rob Browning
Here's the function. According to the profiler, it's called 32094
times, but also according to the profiler, gnc:html-style-table-fetch
is never called, but if I put a print statement just before the set!,
I see that it's called all the time.

Hmm, this is mysterious. It's not at a tail position, and I don't
understand why putting a print statement changes the behavior.
I don't have any idea of how to fix it right now, sorry.

Kei

[Rob Browning]

Post by Keisuke Nishida
Hmm, this is mysterious. It's not at a tail position, and I don't
understand why putting a print statement changes the behavior.
I don't have any idea of how to fix it right now, sorry.

If it helps, I wasn't claiming that putting in the print statement
changed the behavior -- I presume the profiler would still list its
number of calls at 0. I just meant that by putting in the print
statement, I could see that it was in fact being called, even though
the profiler was saying that it wasn't.

Thanks

[Keisuke Nishida]

At 04 Apr 2001 19:47:01 -0500,

Post by Rob Browning

Post by Keisuke Nishida
Hmm, this is mysterious. It's not at a tail position, and I don't
understand why putting a print statement changes the behavior.
I don't have any idea of how to fix it right now, sorry.

If it helps, I wasn't claiming that putting in the print statement
changed the behavior -- I presume the profiler would still list its
number of calls at 0. I just meant that by putting in the print
statement, I could see that it was in fact being called, even though
the profiler was saying that it wasn't.

I see. But I still don't know what is wrong with this.
Could you try with simpler examples? What happens with
the followings, for example:

(define (foo)
(let ((x #f))
(begin (set! x (bar))
(or x x))))

(define (foo)
(let ((x #f))
(begin (bar)
(or x x))))

(define (foo)
(let ((x #f))
(set! x (bar))))

[Rob Browning]

I see. But I still don't know what is wrong with this. Could you
try with simpler examples? What happens with the followings, for

I found at least one thing that's fairly odd, though I don't know if
it has any bearing on the problem we've been discussing. I ran your
tests and the profiler's behavior varies depending on how I run guile.
If you'd like me to run some more tests, just let me know.

$ guile -s test-profiler.scm
Called Procedure Run Real
------ --------- --- ----

0 007 ***@scratch:~/sync/gmc/guile-profiled
$


$ guile -l test-profiler.scm
Called Procedure Run Real
------ --------- --- ----
3 bar 0 0
1 foo-1 1 0
1 foo-2 0 0
1 foo-3 0 0
guile> (quit)

The code from test-profiler.scm:

(use-modules (ice-9 session) (ice-9 format))

(define (dump-profile-data)
(let ((procs (let loop ((vals (map eval (apropos-internal "")))
(procs '()))
(if (null? vals)
procs
(let ((proc (car vals))
(data (object-property (car vals) 'profile-data)))
(if data
(loop (cdr vals) (acons proc data procs))
(loop (cdr vals) procs)))))))
(display "Called Procedure Run Real\n")
(display "------ --------- --- ----\n")
(map (lambda (p)
(let ((proc (car p)) (data (cdr p)))
(format #t "~6a ~32a ~4a ~4a~%"
(vector-ref data 0)
(procedure-name proc)
(vector-ref data 1)
(vector-ref data 2))))
(let ((real (lambda (p) (vector-ref (cdr p) 2)))
(count (lambda (p) (vector-ref (cdr p) 0))))
(sort! procs (lambda (p1 p2)
(if (= (real p1) (real p2))
(> (count p1) (count p2))
(> (real p1) (real p2)))))))))


(define (bar)
#t)

(define (foo-1)
(let ((x #f))
(begin (set! x (bar))
(or x x))))

(define (foo-2)
(let ((x #f))
(begin (bar)
(or x x))))

(define (foo-3)
(let ((x #f))
(set! x (bar))))

(set! *profile-all* #t)
(foo-1)
(foo-2)
(foo-3)
(set! *profile-all* #f)

(dump-profile-data)

[Keisuke Nishida]

At 05 Apr 2001 12:15:04 -0500,

Post by Rob Browning
I found at least one thing that's fairly odd, though I don't know if
it has any bearing on the problem we've been discussing. I ran your
tests and the profiler's behavior varies depending on how I run guile.
If you'd like me to run some more tests, just let me know.
$ guile -s test-profiler.scm
Called Procedure Run Real
------ --------- --- ----
$ guile -l test-profiler.scm
Called Procedure Run Real
------ --------- --- ----
3 bar 0 0
1 foo-1 1 0
1 foo-2 0 0
1 foo-3 0 0

This is because the profiler works only with the debugging evaluator.
With the switch -s, you disable the debugging evaluator by default.
In this case, you need to include (ice-9 debug) explicitly.

[Rob Browning]

Also, from reading your profiling patch, does this code mean that even
if you (set! *profile-all* #f), that it'll keep profiling functions
that already have profile-data?

+ /* Start profiling */
+ prof_data = scm_object_property (proc, sym_profile_data);
+ if (SCM_NFALSEP (SCM_CDR (scm_profile_all)) || SCM_NFALSEP (prof_data))
+ {

If so, I hadn't realized that...

Thanks.

[Keisuke Nishida]

At 05 Apr 2001 12:41:06 -0500,

Post by Rob Browning
Also, from reading your profiling patch, does this code mean that even
if you (set! *profile-all* #f), that it'll keep profiling functions
that already have profile-data?
+ /* Start profiling */
+ prof_data = scm_object_property (proc, sym_profile_data);
+ if (SCM_NFALSEP (SCM_CDR (scm_profile_all)) || SCM_NFALSEP (prof_data))
+ {
If so, I hadn't realized that...

I don't remember what I was trying to do... Maybe I wanted to
profile some selected procedures, instead of all procedures.
*profile-all* is used only when you want to profile all procedures
unconditionally, which may result in inaccurate output (because of
too much overhead).

The patch I posted was just a quick hack; I did not intend it
for practical use. I'm now preparing a better profiling support
for my VM, so I'd rather want to work on it.

Kei

P.S. I've got an idea of how to handle the current macros in my
compiler. Probably I can compile much of existing Scheme code
without changing.

% guile-vm
Guile Scheme interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@guile> (define-macro (foo x) (+ x x))
***@guile> foo
$1 = #<macro 40321f00>
***@guile> (foo 1)
$2 = 2
***@guile>

[Rob Browning]

Post by Keisuke Nishida
The patch I posted was just a quick hack; I did not intend it
for practical use. I'm now preparing a better profiling support
for my VM, so I'd rather want to work on it.

Fair enough. Don't worry about it anymore, then. I've just started
looking at debug.scm, and think I can get a good enough profiler
working via the trace inftrastructur pretty easily. I tried that a
while ago, but got stuck because I misinterpreted something. I think
I understand now, though.

Post by Keisuke Nishida
P.S. I've got an idea of how to handle the current macros in my
compiler. Probably I can compile much of existing Scheme code
without changing.

Interesting. When you get a chance, could you post the latest
instructions for getting your vm installed/working from current CVS?
I'd like to start over from scratch and see if maybe that'll fix my
problems. I may have messed something up during our attempts to fix
the problem.

Thanks.

[Keisuke Nishida]

At 05 Apr 2001 15:56:29 -0500,

Post by Rob Browning
Interesting. When you get a chance, could you post the latest
instructions for getting your vm installed/working from current CVS?
I'd like to start over from scratch and see if maybe that'll fix my
problems. I may have messed something up during our attempts to fix
the problem.

There is a file guile-vm/README, in which there are some instructions.
But I think all you need to do is basically "configure && make install"
in both guile-core/ and guile-vm/. After this, dynamic-link & call
should work..

Can anybody else try my VM if it works?

Thanks,
Keisuke

[Rob Browning]

Post by Keisuke Nishida
There is a file guile-vm/README, in which there are some instructions.
But I think all you need to do is basically "configure && make install"
in both guile-core/ and guile-vm/. After this, dynamic-link & call
should work..

OK, I'll try again from scratch.

Thanks

[Neil Jerram]

Keisuke> Can anybody else try my VM if it works?

I tried, but I don't have SLIB, so I didn't get very far. I'll try
again after installing SLIB.

Regards,
Neil

[Neil Jerram]

Keisuke> Can anybody else try my VM if it works?

Neil> I tried, but I don't have SLIB, so I didn't get very far.
Neil> I'll try again after installing SLIB.

Now with SLIB installed... I'm afraid I still don't get very far:

[***@portalet Downloads]$ guile-vm
;;; Autoloading of compiled code modules is deprecated.
;;; Write a Scheme file instead that uses `dynamic-link' directly.
;;; (You just tried to autoload module (oop goops goopscore).
Standard Scheme (R5RS + syntax-case) interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@user> (+ 1 2)
(#<program 0x812af13>)
ERROR: Unbound variable: +
***@user>

Regards,
Neil

[Keisuke Nishida]

At 06 Apr 2001 15:38:31 +0100,

Post by Neil Jerram
;;; Autoloading of compiled code modules is deprecated.
;;; Write a Scheme file instead that uses `dynamic-link' directly.
;;; (You just tried to autoload module (oop goops goopscore).
Standard Scheme (R5RS + syntax-case) interpreter 0.3 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.
Enter `,help' for help.
(#<program 0x812af13>)
ERROR: Unbound variable: +

This is a very old version. Could you try with the latest version
from CVS?

Anyway, I'll try to release the next snapshot (0.4) soon.

Thanks,
Keisuke

[Neil Jerram]

Keisuke> This is a very old version. Could you try with the
Keisuke> latest version from CVS?

Keisuke> Anyway, I'll try to release the next snapshot (0.4) soon.

With latest CVS, and following through the example session in the
README, I have more success, but fail with `(add 1 2)'.

Note also that guile-vm exits completely if I type `,x +'.

Best regards,
Neil


[***@portalet neil]$ guile-vm
;;; Autoloading of compiled code modules is deprecated.
;;; Write a Scheme file instead that uses `dynamic-link' directly.
;;; (You just tried to autoload module (oop goops goopscore).)
Guile Scheme interpreter 0.4 on Guile 1.4.1
Copyright (C) 2001 Free Software Foundation, Inc.

Enter `,help' for help.
***@guile> (+ 1 2)
$1 = 3
***@guile> ,c -c (+ 1 2)
(@asm (0 0 0 0)
(const 1)
(const 2)
(add 2)
(return 0))
***@guile> ,c (+ 1 2)
Disassembly of bootcode:

Compiled for Guile VM 0.4

nlocs = 0 nexts = 0

0 make-int8:1 ;; 1
1 make-int8 2 ;; 2
3 add
4 return

***@guile> (add 1 2)
ERROR: Unbound variable: add
***@guile> add
ERROR: Unbound variable: add
***@guile> +
$2 = #<primitive-procedure +>
***@guile> ,x +
<unnamed port>: In procedure program-arity in expression (program-arity prog):
<unnamed port>: Wrong type argument in position 1 (expecting PROGRAM_P): #<primitive-procedure +>
[***@portalet neil]$

[Rob Browning]

Post by Neil Jerram
With latest CVS, and following through the example session in the
README, I have more success, but fail with `(add 1 2)'.
Note also that guile-vm exits completely if I type `,x +'.

Hmm. I wonder if that's because + is still a <primitive-procedure>,
not a vm <program>...

[Keisuke Nishida]

At 08 Apr 2001 10:36:53 +0100,

Post by Neil Jerram
With latest CVS, and following through the example session in the
README, I have more success, but fail with `(add 1 2)'.

Oops, there should have been the following expression before it:

(define (add x y) (+ x y))

Post by Neil Jerram
Note also that guile-vm exits completely if I type `,x +'.

My repl does not catch some errors because I want to display
backtrace on errors. I couldn't figure out how to use save-stack
with my repl.

[Neil Jerram]

Keisuke> Can anybody else try my VM if it works?

Neil> I tried, but I don't have SLIB, so I didn't get very far.
Neil> I'll try again after installing SLIB.

I just synched up with CVS, and now I'm having trouble building
guile-vm:

grep '^VM_DEFINE' vm_number.c > vm_number.i
grep: vm_number.c: No such file or directory
make[1]: *** [vm_number.i] Error 2

Full trace is below.

Neil

[***@portalet guile-vm]$ ./autogen.sh
[***@portalet guile-vm]$ ./configure && make
creating cache ./config.cache
checking for a BSD compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking whether make sets ${MAKE}... yes
checking for working aclocal... found
checking for working autoconf... found
checking for working automake... found
checking for working autoheader... found
checking for working makeinfo... found
checking for Guile... yes
checking for gcc... gcc
checking whether the C compiler (gcc ) works... yes
checking whether the C compiler (gcc ) is a cross-compiler... no
checking whether we are using GNU C... yes
checking whether gcc accepts -g... yes
checking whether ln -s works... yes
checking host system type... i586-pc-linux-gnu
checking build system type... i586-pc-linux-gnu
checking for ranlib... ranlib
checking for ld used by GCC... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD-compatible nm... /usr/bin/nm -B
updating cache ./config.cache
checking for object suffix... o
checking for executable suffix... no
checking for gcc option to produce PIC... -fPIC
checking if gcc PIC flag -fPIC works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.lo... yes
checking if gcc supports -fno-rtti -fno-exceptions ... yes
checking if gcc static flag -static works... -static
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the linker (/usr/bin/ld) supports shared libraries... yes
checking command to parse /usr/bin/nm -B output... ok
checking how to hardcode library paths into programs... immediate
checking for /usr/bin/ld option to reload object files... -r
checking dynamic linker characteristics... Linux ld.so
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for objdir... .libs
creating libtool
loading cache ./config.cache
checking labels as values... yes
updating cache ./config.cache
creating ./config.status
creating Makefile
creating src/Makefile
creating doc/Makefile
creating module/Makefile
creating src/config.h
src/config.h is unchanged
Making all in src
make[1]: Entering directory `/home/neil/Guile/cvs/guile-vm/src'
grep '^VM_DEFINE' vm_loader.c > vm_loader.i
guile-snarf envs.c -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 envs.c > envs.x \
|| { rm envs.x; false; }
guile-snarf instructions.c -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 instructions.c > instructions.x \
|| { rm instructions.x; false; }
guile-snarf programs.c -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 programs.c > programs.x \
|| { rm programs.x; false; }
guile-snarf vm.c -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 vm.c > vm.x \
|| { rm vm.x; false; }
cd .. \
&& CONFIG_FILES=src/Makefile CONFIG_HEADERS= /bin/sh ./config.status
creating src/Makefile
make[1]: Leaving directory `/home/neil/Guile/cvs/guile-vm/src'
make[1]: Entering directory `/home/neil/Guile/cvs/guile-vm/src'
/bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 -c envs.c
rm -f .libs/envs.lo
gcc -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 -Wp,-MD,.deps/envs.pp -c envs.c -fPIC -DPIC -o .libs/envs.lo
gcc -DHAVE_CONFIG_H -I. -I. -I. -I/usr/local/include -g -O2 -Wp,-MD,.deps/envs.pp -c envs.c -o envs.o >/dev/null 2>&1
mv -f .libs/envs.lo envs.lo
grep '^VM_DEFINE' vm_number.c > vm_number.i
grep: vm_number.c: No such file or directory
make[1]: *** [vm_number.i] Error 2
make[1]: Leaving directory `/home/neil/Guile/cvs/guile-vm/src'
make: *** [all-recursive] Error 1
[***@portalet guile-vm]$

[Rob Browning]

Post by Neil Jerram
I just synched up with CVS, and now I'm having trouble building
grep '^VM_DEFINE' vm_number.c > vm_number.i
grep: vm_number.c: No such file or directory
make[1]: *** [vm_number.i] Error 2

I had the same problem. It's stale .deps. Just "rm -r src/.deps" and
try again.

[Rob Browning]

Post by Keisuke Nishida
I see. But I still don't know what is wrong with this.
Could you try with simpler examples? What happens with

OK, I've investigated further and here's a more conclusive example
that something's wrong, and now I'm wondering if it could either be a
namespace issue, or perhaps more likely, there's a case in eval.c
that's not being handled.

In this example, we're profiling report generation, and so in the
top-level reporting function we do this (this is the only time
*profile-all* is touched in the app):

(set! *profile-all* #t)
(set! formlist (gnc:html-document-render doc))
(set! *profile-all* #f)
(dump-profiling-data)

Then over in gnc:html-style-table-fetch, I do this in an
unconditionally executed part of the body:

(if *profile-all*
(begin
(display "inside gnc:html-style-table-fetch\n")
(display " *profile-all* ") (display *profile-all*) (newline)
(display " ")
(display (procedure-properties gnc:html-style-table-fetch))
(newline)))

and after running one trivial report, the resulting output looks like
this (note that 'profile-data is not set, which should be impossible
if I'm reading the intent of your profiling patch correctly):

inside gnc:html-style-table-fetch
*profile-all* #t
((arity 3 0 #f) (name . gnc:html-style-table-fetch))
inside gnc:html-style-table-fetch
*profile-all* #t
((arity 3 0 #f) (name . gnc:html-style-table-fetch))

[... many duplicates omitted ...]

inside gnc:html-style-table-fetch
*profile-all* #t
((arity 3 0 #f) (name . gnc:html-style-table-fetch))
inside gnc:html-style-table-fetch
*profile-all* #t
((arity 3 0 #f) (name . gnc:html-style-table-fetch))
inside gnc:html-style-table-fetch
*profile-all* #t
((arity 3 0 #f) (name . gnc:html-style-table-fetch))

and we get a profile dump like this (note that
gnc:html-style-table-fetch is listed as never called):

Called Procedure Run Real
------ --------- --- ----
1197 hash-fold 73 77
3208 eval 52 49
1389 root-module-closure 51 49
37 apply 34 36
1214 for-each 33 35
4365 eq? 13 16
7757 not 6 14
1 gnc:html-document-render 8 7
1322 = 4 6
1 gnc:html-table-render 5 4
4418 vector-ref 4 3
4643 struct? 3 2
4383 hashq-set! 3 2
7623 cdr 6 5
3452 null? 0 2
1408 symbol-interned? 2 2
3 gnc:html-text-render 2 2
4087 struct-ref 2 2
3363 cons 2 1
1198 symbol? 0 1
242 display 2 2
38 make-struct 0 1
28 gnc:html-style-info-merge 0 1
20 hash-ref 0 1
14 gnc:html-document-markup-start 1 1
7 gnc:html-document-fetch-data-style 1 1
3 gnc:html-document-fetch-markup-style 0 1
1 gnc:html-table-cell-render 1 1
6706 car 1 0
6710 car 1 0
4341 struct-vtable 3 0
1408 variable-bound? 2 0
1408 builtin-variable 0 0
1177 variable? 0 0
200 acons 0 0
148 gnc:html-markup-style-info? 0 0
129 gnc:html-data-style-info? 0 0
106 assoc-set! 0 0
106 vector-set! 0 0
98 assoc-ref 0 0
3145 >= 3 3
64 newline 0 0
57 gnc:html-style-table-compiled? 0 0
5293 + 4 1
51 struct-set! 0 0
42 eqv? 0 0
42 make-vector 0 0
40 list? 0 0
39 apply:nconc2last 0 0
283 string? 0 0
32 procedure-properties 0 0
29 gnc:html-data-style-info-merge 0 0
24 gnc:html-markup-style-info-merge 0 0
1243 string-ref 0 1
22 gnc:make-html-markup-style-info 0 0
18 string-set! 0 0
16 char? 0 0
529 string-length 0 0
538 string-length 0 0
15 cadr 0 0
15 cadr 0 0
15 gnc:html-table-cell? 0 0
13 pair? 0 0
453 length 0 0
11 gnc:html-document-markup-end 1 0
271 number? 0 0
538 - 0 1
545 list 1 1
8 cddr 0 0
8 gnc:make-html-object 0 0
7 min 0 0
7 gnc:default-html-string-renderer 0 0
7 procedure? 0 0
6 negative? 0 0
5 gnc:html-object? 0 0
5 record? 0 0
233 number->string 0 0
928 memq 2 1
3 truncate 0 0
3 inexact->exact 0 0
3 * 0 0
3 / 0 0
330 substring 2 0
334 append 3 1
2 vector? 0 0
512 boolean? 0 0
2 make-string 0 0
2 equal? 0 0
442 char-numeric? 0 0
2 require:require 0 0
2 require:provided? 0 0
2 require:require 0 0
1 reverse 0 0
1 gnc:html-markup/no-end 0 0
1 string=? 0 0
1 caddr 0 0
1 string-append 0 0
1 caddr 0 0
0 gnc:make-html-document 0 0
0 gnc:html-markup-style-info-attributes 0 0
0 gnc:html-style-table-primary 0 0
0 gnc:html-document-set-style-internal! 0 0
0 gnc:html-document-style-sheet 0 0
0 gnc:html-object-renderer 0 0
0 gnc:html-style-sheet-options 0 0
0 gnc:html-style-table-inheritable 0 0
0 gnc:make-html-markup-style-info-internal 0 0
0 record-type-fields 0 0
0 stdio:sprintf 0 0
0 module-obarray-set! 0 0
0 gnc:html-data-style-info-renderer 0 0
0 gnc:html-object-data 0 0
0 module-modified 0 0
0 gnc:html-document-push-style 0 0
0 gnc:option-data 0 0
0 kvt-ref 0 0
0 gnc:option-getter 0 0
0 gnc:html-data-style-info-inheritable? 0 0
0 record-type-descriptor 0 0
0 gnc:html-table-col-style 0 0
0 gnc:html-table-row-style 0 0
0 gnc:lookup-option 0 0
0 gnc:html-document-append-objects! 0 0
0 gnc:html-markup-style-info-inheritable? 0 0
0 gnc:make-html-text-internal 0 0
0 module-add! 0 0
0 gnc:make-html-document-internal 0 0
0 gnc:html-document-render-data 0 0
0 gnc:html-document-style-stack 0 0
0 record-modifier 0 0
0 gnc:html-table? 0 0
0 hash->kv-list 0 0
0 gnc:make-html-text 0 0
0 gnc:html-style-table-set! 0 0
0 gnc:html-table-col-styles 0 0
0 gnc:html-table-row-markup 0 0
0 gnc:html-table-row-styles 0 0
0 gnc:html-table-cell-style 0 0
0 gnc:html-text-style 0 0
0 gnc:html-table-cell-data 0 0
0 gnc:html-table-row-markup-table 0 0
0 gnc:html-table-style 0 0
0 gnc:html-object-render 0 0
0 gnc:html-document-objects 0 0
0 gnc:option-value 0 0
0 gnc:html-style-table-set-compiled! 0 0
0 gnc:make-html-style-table 0 0
0 gnc:html-markup-style-info-font-size 0 0
0 gnc:html-markup-style-info-font-face 0 0
0 gnc:html-document-set-objects! 0 0
0 gnc:color-option->html 0 0
0 module-observers 0 0
0 gnc:html-markup-style-info-closing-font-tag 0 0
0 gnc:html-style-sheet-renderer 0 0
0 gnc:html-document-set-style! 0 0
0 gnc:html-table-col-headers 0 0
0 gnc:html-table-col-headers-style 0 0
0 gnc:color->html 0 0
0 gnc:html-style-table-set-inheritable! 0 0
0 gnc:html-markup-h3 0 0
0 gnc:html-markup-style-info-font-color 0 0
0 gnc:html-document-style 0 0
0 gnc:html-document-title 0 0
0 gnc:html-table-data 0 0
0 gnc:html-style-sheet-style 0 0
0 for-each-in-order 0 0
0 gnc:html-document-pop-style 0 0
0 gnc:html-document-add-object! 0 0
0 module-weak-observers 0 0
0 module-obarray 0 0
0 gnc:html-markup-p 0 0
0 gnc:make-html-style-table-internal 0 0
0 gnc:html-table-cell-tag 0 0
0 list-index 0 0
0 gnc:html-table-cell-colspan 0 0
0 gnc:html-table-cell-rowspan 0 0
0 gnc:html-markup-style-info-tag 0 0
0 kvt-set! 0 0
0 gnc:html-table-caption 0 0
0 gnc:html-data-style-info-data 0 0
0 gnc:html-text? 0 0
0 gnc:html-style-table-uncompile 0 0
0 gnc:html-style-table-compiled 0 0
0 gnc:html-document-set-style-stack! 0 0
0 record-type? 0 0
0 gnc:html-text-body 0 0
0 gnc:html-style-sheet-render 0 0
0 gnc:make-html-object-internal 0 0
0 make-kvtable 0 0
0 gnc:html-style-table-fetch 0 0
0 gnc:html-style-table-compile 0 0
0 kvt-fold 0 0
0 gnc:html-markup-style-info-set-attribute! 0 0
0 stdio:sprintf 0 0
0 stdio:sprintf 0 0
0 generic-write 0 0
total time to run report: 2.343889